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PREFACE 



This study, produced at the request of the National Center for Research on Vocational 
Education (NCRVE), eicplores education accountability in the United States. More specifi- 
cally, its focus is accountability in vocational education. The study began as an examination 
of the potential uses for vocational education performance information by Federal and state 
oflBdals, local educators, employers, parents, and students. It turned quickly, however, to an 
examination of two prior issues. First, it considers the way states and localities are coping 
with changes in state administration reqmred by P. L. 101-392 (The Carl D. Perkins 
Vocational and Applied Technology Amendments of 1990), as well as their capabilities to ad- 
vance accountability through performance standards. The second issue addressed concerns a 
question raised repeatedly during state and local interviews, asking whether the Perkins 
amendments imply the need to develop statewide aggregate measures of program perfor- 
mance and, ultimately, national aggregate measures of performance as the foundation of ed- 
ucational accountability. 
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SUMMARY 



This Report is the result of a study of accoimtability in vocational education, conducted by 
RAND for the National Center for Research in Vocational Education (NCRVE). The study 
had two goals: First, to understand the intentions of the people who formulated and enacted 
the new requirements for vocational education performance standards and accoimtability 
continued in the Carl Perkins Act; second, to understand how those requirements would af- 
fect state and local accountability processes in vocational education. The study focused on 
the "to whom" side of accoimtability, trying to determine who needed evidence about the per- 
formance of vocational education and how they would use that evidence. NCRVE saw the 
project as a companion to other activities that examined the *Tor what" of vocational educa- 
tion accountability, considering the measurement implications of the Perkins Act require- 
ments on performance standards and measures. The project consisted of a set of interviews 
and case studies at the Federal, state, and local levels about performance information on vo- 
cational education: who needs it, what should it look like, and how will it be used? Case 
study states included Maryland, Pennsylvania, California, Ohio, New York, Oklahoma, and 
Indiana. 

The message that emerged from the research is that the main focus of accountability in 
vocational education is local, and that the Perkins Act requirements for new performance 
measures and standards are meant to strengthen local accountability, not to weaken it or to 
create new channels of reporting to the Federal government. 

PURPOSE OF THE REQUIREMENTS 

Interviews with Congressional staff members and lobbyists made it clear that no one in- 
tended to create a flow of information to Washington that would let Congress or anyone else 
micromanage vocational education. Nor did the drafters of the act hope to obtain a master 
data base from which they could derive an unambiguous assessment of vocational education 
nationwide, or in any state or locality. 

It was clear, however, that members of Congress and senior committee staff were worried 
about the quality of local programs and were uncertain about whether they were being im- 
plemented with due concern for the needs of local students and employers. This concern was 
based on business leaders' complaints about the quality of high-school graduates (including, 
but not specific to, graduates of vocational programs), and on Congressional members' own 
fears that some local managers of vocational education had become bureaucratic, deadened 
by routine, and more concerned with protecting staff jobs than with meeting local labor mar- 
ket needs. 

The Perkins Act requirements were meant to reassure critics that the legislative supporters 
of vocational education were serious about quality, and that lax or self-indulgent program 
administration would not be tolerated. But members of Congress did not want to set them- 
selves up as the remote, ultimate judges of local processes. Instead, they wanted to create a 
framework of processes that would guarantee that quality issues received due attention at 
the local and state levels. 
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The drafters of the Perkins requirements have decades of experience with Title I/Chapter I. 
Thus, they did not intend to stimulate another futile search for a national bottom-line as- 
sessment of a program that is, by design, local in its goals, services, and outcomes. To the 
contrary, they intended to set up processes that would make local program performance 
manifest to local stakeholders. 

EFFECTS ON STATE AND LOCAL ACCOUNTABILITY 

Through case studies in seven states, including both urban and rural localities, local ac- 
countability processes were found to vary in quality. In some localities, the quality and rele- 
vance of vocational education programs is ensured through such processes as the following: 

• Employer advisory committees 

• Tracking of employer needs through student placement 

• Attracting students via job placement success 

• Employer certification of the need for new courses 

• Eliminating or upgrading courses with low enrollment or poor placement records. 

However, many local processes have serious weaknesses, including a **routine following^ 
mentahty among vocational education providers and lax tracking of student placement. 
Further faults include low expectations for student skill attainment and job placement, little 
capacity to project future labor force needs, and inattention to the needs of transient, low-in- 
come minority students. 

There are also real conflicts of interest among local constituencies. In very simple communi- 
ties with only a few employers, it is possible for healthy accountabihty processes to work in 
everyone's interests. The range of employment opportunities is finite and generally under- 
standable, and all employers can take part in the normal processes of consultation. In larger 
communities, however, only a fraction of employers can be consulted directly. Larger and 
longer-established employers naturally are more hkely to be consulted than very small or 
new employers. In some places this means that important employer interests— the small 
firms that normally employ new high school graduates and the newer firms that may rely on 
imfamiliar technologies and offer unconventional career opportunities— may be poorly repre- 
sented. 

Conflicts of interest among students and employers also may arise. Established employers 
may want students prepared for traditional low- and moderate-skilled jobs, which might 
block students from obtaining jobs with other employers, possibly in other localities. 

These conflicts of interest justify oversight by state and federal governments. The state 
needs to protect its broader economic future against the narrow localism and short-time hori- 
zon that can dominate local accoumtability processes. The state and federal governments 
need to help disadvantaged groups that lack the leverage to protect themselves at the local 
level. 

Local actors' horizons can be expanded in several ways. Information about future economic 
trends can help employers to anticipate their own needs. Information about the needs of 
broader regional, statewide, or national labor markets also can help educators and students 
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iinderstand opportunities that might not be evident locally. Lists of competencies required 
by advanced employers— and testing systems to help students and educators assess their 
performance in light of those needs— can create demands for program improvement 

Competency-based systems such as those being built in Oklahoma can both raise standards 
and provide concrete guides to program improvement. Such systems derive from lists of 
needed competencies created by knowledgeable employers. When these lists are used as the 
basis for curriculum development, student testing, and certification, they guide the voca- 
tional education policy and build employers' competence, as well. Also, they provide external 
standards against which program quality and student access can be evaluated. 

Based on our case studies, however, we concluded that hard outcomes data should supple- 
ment, not supplant, processes of consultation and monitoring and the use of managerial 
judgment. 

All educational outcomes are products of multiple causation. Test scores, graduation rates, 
dropout rates, and student placement results can be affected strongly by other factors inde- 
pendent of the quality of training and instruction students have received. Students experi- 
encing a given vocational program may have different change scores, depending on the qual- 
ity of education they receive outside vocational education. 

The outcomes of greatest interest in vocational education also are affected by local labor 
market conditions. Vocational education students' job placement rates, hours of work, in- 
come, and rates of career progression can be affected profoundly by the demand for particular 
kinds of labor, which, in turn, is determined by the current level and composition of local 
economic activity. 



CONCLUSION ON NATIONAL DATA SYSTEMS AND ACCOUNTABILITY 

These complications do not rule out the use of hard outcomes data. However, they do illus- 
trate the many ways in which such data can foster misunderstanding of program perfor- 
mance. Outcomes data must be interpreted in light of other information about student 
characteristics and prior educational preparation, the quality of educational programs that 
supplement vocational education, and local labor market conditions. Without such con- 
textual information, vocational educators are forced to do everything possible to raise their 
outcome scores. This promotes "gaming" to increase scores (e.g., by '^creaming" the student 
population, or by finding ways to discount the outcomes of less able students). Also, it mis- 
leads policymakers, frequently causing them to castigate local providers whose programs are, 
in fact, performing reasonably well in light of local conditions. 

During a period of fiscal stringency, many states are tempted to cut their staffs and to aban- 
don direct oversight in favor of automated systems of measurement. These systems can pro- 
vide clear comparative information about the priorities and average outcomes attained by 
different localities and programs. But, as suggested above, hard outcomes data alone can 
lead to false attributions of causality and can mislead policymakers. State and federal offi- 
cials may overlook crucial local conditions and reward or punish programs whose outcomes 
are largely determined, for good or ill, by uncontrollable local circumstances. 

When faced with complaints about system failure, top-level policymakers (Congress, state 
legislatures, presidents, governors) demand accountability. That is, they seek systems ac- 
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coxintability-Hiot separate auditing of the effects of differfmt programs. If the response to 
the call for accountability is to further separate educational programs from one another, 
system performance will continue to decline, with increasingly negative consequences for vo- 
cational education's reputation and public support. 

The bottom line is that vocational education must become an integral part of the local educa- 
tional system, and its accountability system muist emphasize meeting local needs. 
Standardized outcome measures can strengthen the local accountability process if they give 
local parents, employers, and commimity leaders objective information about local program 
outcomes. However, if they are used in ways that weaken local accoxintability, standardized 
outcome measures ultimately can impair local program performance and damage further the 
local and national reputation of vocational education. 

In general, accountability driven by a national data base on student outcomes does not solve 
the problems of ensuring equity and quality in vocational education or other federal pro- 
grams. In addition, such accoimtability can threaten local performance. Efforts to automate 
accoimtability via a national data system are counter-productive. Federal and state officials 
should strengthen local accountability processes, not compete with them. The best way to 
avoid the negative consequences of a national performance data base is to abandon any effort 
to create a national data base on student performance. 
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1. PANDORA^SBOX 



Senator Robert F. Kennedy was the father of program accountability in education. When, in 
1965, he created a mysterious black box known as **Federal education evaluation," he could 
not have known that it quickly would take on a life of its own, growing beyond its creator s 
expectations and control. In fact, accountability schemes often have proved since to be more 
trouble than they were worth. But like Pandora^s box, which unleashed misery on an imsus- 
pecting world while continuing to shelter the virtue of hope, the black box of evaluation and 
accoimtabihty still holds some promise. The task of pohcymakers and analysts is to find 
ways of harvesting the power and potential of accoimtability, while stuffing its shortcomings 
and potential for mischief back into the box. 

It is safe to say that Kennedy's idea that federal prograjns should be formally evaluated has 
helped to transform education administration and accountability in the United States. Until 
the mid-1960s. Federal oversight of American education consisted of little more than collect- 
ing data on enrollments and the number of diplomas and degrees r warded annually.^ The 
idea that the Federal government would insist on monitoring program quality, institutional 
performance, or student achievement was virtually inconceivable. 

But times have changed. Since 1965, the Federal government has moimted several evalua- 
tions of Chapter 1 of the Elementary and Secondary Education Act (formerly Title I), three 
separate assessments of the Carl D. Perkins Act (formerly the Vocational Education Act), 
and periodic assessments of the skills of 9-, 13-, and 17-year olds by the National Assessment 
of Educational Progress (NAEP), and has cooperated in supporting international assess- 
ments of student achievement. The Secretary of Labor's Commission on Achieving Necessary 
Skills (SCANS) reviewed the preparation of American youth and proposed a national agenda 
to ensure student workplace readiness; the President of the United States and the Secretary 
of Education call for "American Achievement Tests'* in five key areas of curriculum; and 
Congressionally mandated panels on education indicators, and on education standards and 
testing, issue analyses calling, in effect, for a better national definition of education goals and 
more accurate assessments of their achievement.^ In short, by 1991, the strong views of an 
earlier generation opposed to Federal assessment have been stood on their heads. 

Against this backdrop, what, precisely, do the new accountability provisions in Perkins 
•mean? That is, what does the term itself signify and what did the framers of the legislation 
hope for? How do vocational education administrators at the state and local levels under- 
stand these provisions and how well prepared are they to act on them? Finally, what would a 
reasonable breakdown of responsibilities among Federal, state, and local administrators look 
like in a useful performance-based system of accountability in vocational education? 

Succeeding sections of this Report take up these questions. This section provides a brief 
overview of the history of evaluation activity in Federal education programs and sets the ma- 



^Richard J. Mumane, Improving Educational Indicators and Economic Indicators: The Same Problem," 
Educational Evaluation and Policy Analysis (Summer 1970). 

^See, for example, Education Counts: An Indicator System to Monitor the Nation's Educational Health 
(Washington^ D.C., National Center for Education Statistics, July 1991). 
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jor accountability, evaluation, and performance provisions of the 1990 Perkins legislation in 
that context* Section 2 presents an analytical fraraework for understanding accountability 
and performance standards. It aims to illuminate the relationship between performance 
measures (specific ways of registering the results of vocational education) and education ac- 
countability. It attempts to clarify the terms of the discussion in the belief that lack of clarity 
in the past has helped to confuse hopelessly the conversation among Federal, state, and local 
officials and the general public. 

Section 3 draws on interviews with state and local officials to describe how they understand 
the new requirements and to assess their organizational capacity to implement them. It 
should come as no surprise to learn that state responses vary along several dimensions. 
Some states have almost no concern about the new requirements, believing that their own 
well-developed accountability schemes can respond to any new requirements from 
Washington. Others are desperately concerned that they lack the capabilities to respond. 
Some stete officials, armed with the support of legislative allies, enjoy considerable discre- 
tionary funding to expand and modify their accountability mechanisms. But colleagues in 
other jurisdictions are scrambling with the administrative chaos of staff and budget reduc- 
tions forced by state deficits and reductions in Federal support for state administration. 

Conceptually, the greatest problem lies in radically different understandings at both the 
state and local levels about the meaning of "performance standards.'' Some officials are con- 
vinced performance standards imply pre- and posttesting of all vocational education stu- 
dents. Others hold strongly to the belief that such a view is a mistake and that the perfor- 
mance standards requirements are best understood as a mandate to enhance local program 
improvement efforts and local accountability to students, parents, and employers. 

Section 4 draws on interviews with Federal officials, in Congress and the U.S. Department of 
Education, to explicate their understanding of these new provisions. Section 5 seeks to pull 
all of this together with a summary of the implications of this study for accountability m vo- 
cational education and a recommendation that federal and state accountability efforts focus 
on local program improvement, not on creation of a national data base. The Federal role in 
this scheme becomes one of encouraging states to take local needs into account; that is, the 
state role is one of monitoring local procedures and providing advice and technical assistance. 



EDUCATION ACCOUNTABILITY AND THE PERKINS AMENDMENTS 

What was Robert Kennedy trying to accomplish when he added an evaluation requirement to 
the legislation creating Title I of the Elementary and Secondary Education Act of 1965? 
When President Lyndon B. Johnson proposed the legislation, he demanded that Congress act 
on it promptly and without amendment. The bill swept through the House and Senate to 
Johnson's desk for signature in about 100 days. Kennedy's amendment, added on the Senate 
floor, was the only change to the President's original proposal, and it was adopted without 
committee hearings and with little consideration of its intent or its effects. The Senator's 
floor statement, therefore, essentially is the only legislative history available on his intent, 
and he made his objective quite clear. 

Concerned that Federal education money would disappear without a trace into schools' gen- 
eral operating budgets, Kennedy proposed his amendment, as he explained, to ensure that 
local parents and citizens would know how well they were being served by Federal money 



3 



The initial statute, and its regulations, imposed very few constraints on how schools used the 
Federal money. Since the funds were intended to benefit districts and schools with many 
low-income children, Keimedy was intent on making sure that low-income parents could pro- 
tect their children's interests. Using the data provided by the evaluation requirements, par- 
ents could, if necessary, confront local educators and press for program improvements. 

Almost immediately, the local focus of the evaluation requirements became obscured. Armed 
with data generated locally in response to the Kennedy amendment, the NAACP Legal 
Defense developed a national analysis of inappropriate use of the funds at the local level.^ 
Replete with shocking stories of the use of Federal monies for local tax relief, for band imi- 
forms, for racially segregated white schools, and for athletic equipment, the report had two 
effects. First, in extending the Elementary and Secondary Education Act (ESEA) in 1967, 
Congress began to limit school district flexibihty in the use of Title I funds. The second effect 
was more significant in light of the current study: In order to fend off critics opposed to 
Title I, Congress asked the U.S. Office of Education (the precursor to the current U.S. 
Department of Education) for a nationwide evaluation of the effectiveness of Title I. 

Title I, and federal education evaluation, spent 20 years recovering from the effects of that 
simple request. The request for an analysis of the educational effects of Federal spending 
was practically impossible to meet for two reasons. First, local programs were too variable in 
intent and design to be evaluated only via reading and mathematics tests. Second, the spe- 
cific consequences of federal funds could not be disentangled from the results of a much 
larger state and local effort. Analysts turned to what appeared to be the most promising 
technology available to them— pre- and posttesting of Title I students— and aggregated the 
results to the national level With the benefit of hindsight, the results were predictable: 
Researchers, try as they might, were unable to demonstrate any unambiguous educational 
effects for students participating in Title I programs. Although the same researchers 
protested that their findings did not prove the lack of educational effects (but simply that ef- 
fects could not be validly measured with available methods), critics of the program took the 
results as evidence that Title I did not "work." 

Subsequent analyses, in fact, demonstrated that Title I worked quite well on its own terms.^ 
It "worked" in the ways that its framers had intended: 

• It delivered funds to school districts with high proportions of children from low-income 
families. 

• Within school districts, it delivered funds to schools with the highest proportion of lc\;- 
income students. 

• It purchased additional educational services for eligible children; in particular, it ex- 
panded the amount of time spent on such basics as reading and mathematics. 

• Disadvantaged students who received these services (and benefited from the greater state 
and local attention to compensatory education stimulated by the federal requirements) 
made substantial achievement gains in the basic skills. 



^NAACP Legal Defense Fund, ESEA Title Ir Is It Helping Poor Children^ (1972). 

r^/r^^'/l?f ®^*™P^®» Evaluating Compensatory Education. Washington, D.C., 1976; Launor Carter, The Sustainifw 
hffects of Compensatory Education, Systems Development Corporation. SanU Monica. California, 1978. 
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But tho educational effects were demonstrable only after asking and answering a key se- 
quence of questions grounded in the legislative intent of Title I: Does the money go where it 
is supposed to go? Is it properly and efficiently administered? Does it purchase additional 
educational services? Only after answering these questions affirmatively did the question 
asked first by most evaluators make sense: Do participating children show any gains? 

As the capability to analyze the effectiveness of Federal program expenditures developed, the 
key realization was that the effort to aggregate achievement data across all participants was 
doomed to failure. Aggregate data concealed as much as they revealed for two reasons: first, 
they overlooked critical differences in local program goals and services; and second, they ob- 
scured the fact that some local programs were performing well and others were in grievous 
need of improvement. 

ACCOUNTABILITY AND THE PERKINS ACT 

It is against that backgroimd that the performance standards requirements of the 1990 
Perkins vocational education amendments should be understood. The performance stan- 
dards requirements are, in fact, one of several accountability provisions set forth in different 
sections of the statute. Congressional sophistication in matters of evaluation has grown 
enormously since 1965 and 1967, and the Title I mistake need not be repeated. That 
sophistication is demonstrated in the language of the recent statute, since different sections 
select various pieces of the accountability puzzle and request a distinctive response.^ Taken 
together, the Perkins Act accountability requirements set up state and local processes fo- 
cused on providing high-quality programs to meet distinctive local needs. They emphasize 
adaptability and sustained improvement efforts and call for a strong state role in assisting 
troubled programs. They de-emphasize the creation of standard data bases for national per- 
formance standards. 

Since the late 1970s this logic has characterized the evaluation of major federal education 
programs. In each successive reauthorization of tho Elementary and Secondary Education 
Act, the Vocational Education Act, and the Education for All Handicapped Children Act (PL 
94-142), Congress has requested fiscal, administrative, and student outcome studies from 
multiple sources. It has displayed an increasingly sophisticated concern with the complex 
processes by which a federal grant program is translated into state and local administrative 
and program delivery, and, ultimately, into services and benefits for students. 

As a longtime House staffer concerned with higher education told ub, the Perkins account- 
ability requirements were seen as the most sophisticated approach to evaluation ever taken 
by Congress. They require data and analysis on all aspe.^*;s of vocational education imple- 
mentation and they place responsibility for program impn 'ements squarely in the laps of 
the states. 



^Conference Report (to accompany H Jl. 7), 'Tlie Carl D. Perkins Vocational and Applied Technology Education 
Act Amendments of 1990." Washington, D.C., House of Representatives, 101st Congress, 2nd Session. August 2 
1990. (Beport 101-660). 
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National Assessment of Vocational Education 

Section 403 of Perkins, as amended, requests a comprehensive assessment of vocational edu- 
cation programs. Significantly, the requirement is for an ^assessment" of ^vocational educa* 
tion programs assisted under this Act" — ^not for an "^evaluation" of educational effects. 
Section 403 specifies ten areas of particular interest to the Congressional sponsors: 

• Effects on administration, including state, tribal, and local administration of vocational 
education programs; 

• Expenditures at the Federal, state, and local levels to address program improvement in 
vocational education; 

• Teacher preparation and qualifications; 

• Access of special populations to vocational education programs of high quality; 

• Academic and employment outcomes, including the effects of the education reform move- 
ment on vocational education programs, integration of academic and vocational curricula, 
school-to-work transition, and the relationship of vocational training to subsequent em- 
ployment; 

• Employer involvement in, and satisfaction with, vocational programs; 

• Effect of performance standards and other measures of accountability on the delivery of 
services; 

• Effect of Federal requirements specifying criteria for services to special populations, par- 
ticipatory planning in states, and articulation between secondary and postsecondary pro- 
grams; 

• Coordination with the Job Training and Partnership Act (JTPA) and other Federal pro- 
grams; and 

• Involvement of minority students in vocational student organizations. 
National Center on Vocational Education 

Section 404 of Perkins; as amended, sketches the broad outlines of a vocational education re- 
search agenda for the National Center for Research in Vocational Education (NCRVE). Once 
again, the Act does not request an achievement/outcomes-oriented array of studies, but 
rather a variety of descriptive studies designed to provide information on how vocational ed- 
ucation programs "look and feeF at the local level. In particular, the activities should include 
investigations of: 

• Economic changes affecting the skills employers need and entrepreneurs seek; 

• Integrr.cion of academic and vocational education; 

• Methods for delivering vocational education; 

• Articulation between school and college programs offering high-quality work experience; 

• Recruitment, education, and enhancement of vocational teachers; 

• Accountability processes, including identification and evaluation of performance standards 
for student, program, and state-level outcomes; 



• Education in all aspects of the industry for which students are preparing; 

• Provision of literacy and other communications skills for effective job performance; 

• Identification of high-priority skills and **skill formation approaches'* needed to maintain 
American competitiveness; and 

• Approaches to developing entrepreneurial opportunities for minority-owned enterprises. 
State Plans and Their Approval 

Sections 113 and 114 of Perkins, as amended, govern the plans and approval criteria re- 
quired of states seeking funds under the Act. In general, these provisions require states to 
show that they and their local agents have exercised due care in designing and implementing 
vocational education programs. State plans must include basic information across a wide 
spectrum of program activities, including: (1) provisions for needs assessment; (2) assur- 
ances that criteria for special populations will be met; (3) descriptions of funds allocation for 
basic programs and for special populations; (4) use of funds for occupationally specific 
training, including labor market analyses; (5) the development of perforrrxince standards and 
performance measures (emphasis added); (6) assurances that relevant training and education 
will be provided to men and women in nontraditional fields; and (7) use of performance eval- 
uations with eligible recipients (emphasis added). 

State and Local Standards and Measures 

Section 115 addresses the performance standards that lie at the heart of this study. It stipu- 
lates that each state board must develop and implement a statewide system of **core stan- 
dards and measures of performance" for secondary and postsecondary vocational education 
programs, developed in consultation with a broadly representative State Committee on 
Practitioners. The statewide system must include: 

• Measures of learning and competency gains; 

• One or more measures of performance chosen from competency attainment, job or work 
skill attainment, retention in school or completion of a diploma; and placement in addi- 
tional training, military service, or employment; 

• Incentives to encourage services to special populations; and 

• Procedures for using existing resources and methods developed in other programs receiv- 
ing Federal assistance. 

In developing the system and working with the State Committee on Practitioners, each State 
Board is required to provide the committee with information on different types of standards 
and measures, including an analysis of the advantages and disadvantages of each type, and 
examples demonstrating when such standards and measures have been effective and ineffec- 
tive. 

Finally, the Secretary of Education is required to provide technical assistance to the states in 
developing these systems and to provide, in four years, a report to Congress which will in- 
clude a detailed description of the status of each state's system, will assess the validity, pre- 
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dictiveness, and reliability of standards and measures, and will evaluate the comparability of 
state-developed performance standards across states to establish a **core of common indica- 
tors.** 

Assessment^ Evaluation, and Improvement 

Sections 116 and 117 roimd out this portfolio of accountability provisions by prescribing a 
broad general framework for state and local collaboration in program assessment and im- 
provement Section 116 amends Perkins to reqiiire state assessment of program quality. 
The criteria for assessment should be as broad as the goals of vocational education and 
should include such factors as (1) integration of academic and vocational curricula; (2) se- 
quential course of study leading both to academic and occupational competencies; (3) in- 
creased student work skill attainment and job placement; (4) improved linkages i^etween 
secondary and postsecondary educational institutions; (5) instruction and experience in all 
aspects of an industry; (6) ability of eligible recipients to meet the needs of special popu- 
lations; (7) improving pro^nram quality in schools with high concentrations of poor and low- 
achieving students; and (8) program relevance to the workplace and to future labor market 
needs. 

Section 117 addresses evaluation and improvement at the local service level (secondary, 
postsecondary, and adult education programs). Local programs are called upon to evaluate 
annually the effectiveness of their programs, assessed against the standards and measures 
developed by the state. After one year, if this evaluation indicates that local programs are 
not making substantial progress toward meeting the state's standards and performance mea- 
sures, local administrators are directed to develop a plan in consultation with teachers, par- 
ents, and students to describe how the standards and measures can be met. 

Failure at this second stage invites the state into the process. States are directed to work 
with localities (and their clients) to develop a plan for program improvement if, after one 
year, the locally developed plan has made insufficient progress. The state is to help develop 
a description of the technical assistance it will provide, a reasonable timetable to improve 
school performance, and a description of strategies to be employed. The state is directed to 
continue this eff'ort in consecutive years imtil the local program attains the state and local 
standards and measures developed under Section 115. 

These, then, are the major provisions relating to accoimtability, assessment, and perfor- 
mance standards in vocational education contained in the 1990 statute. They were accom- 
panied by another change in the legislation that represents a significant reshaping of the 
Federal government's relationship with state and local vocational education administrators. 
In an effort to direct more funds to local programs, Perkins' allocations for state administra- 
tive expenses have been reduced significantly. 

The sum of all these changes is a major shift in Federal thinking about the appropriate em- 
phasis at the state and local levels. States, in effect, are being encouraged to refocus their 
activities from curriculum development and technical assistance to monitoring and improv- 
ing local program quality through needs assessment, planning, setting performance stan- 
dards, and evaluation. At the same time, state administrators are being asked to help local 
vocational program directors as they try to expand access of special populations to high- 
quality programs, to promote sex equity, to integrate academic and vocational curricula, to 
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expand efforts to educate students in ''all aspects of industry," and to improve articulation 
between secondary and postsecondary programs. 

CONCLUSIONS 

As this brief overview of the history of Federal evaluation and the principal provisions of 
Perkins demonstrates, Congressional education experts have taken advantage of the experi* 
ence gained in evaluating earlier Federal efforts by asking various agents to take responsibil- 
ity for various aspects of the vocational education accountability puzzle. A National 
Assessment of Vocational Education is directed to provide an overview of program success by 
examining such issues as administration, expenditures, access, and academic and employ* 
ment outcomes (including the effects of education reform), NCRVE has been asked to mount 
research on similar kinds of questions, including accountability processes and performance 
standards.^ States are directed to develop and implement systems of standards and mea- 
sures of performance and are offered several measures of performance from which to choose. 
Further, states are directed to assess local program quality, and local programs are required 
to evaluate the etfectiveness of their programs annually against the standards and measures 
developed by the state. 

Given these several different directives to different agents, Congress clearly is not interested 
in a single measure of program performance as a bottom-line summary of the effectiveness of 
vocational education. For its own purposes, it mandates a complex and careful national as- 
sessment. It tries to enhance state and local accoiintability processes, as well, and to estab- 
lish a clear state responsibility for assistance to troubled vocational education programs. 
This demand for many complementary assessment and accountability processes indicates 
that Congress wants broadly available information about program outcomes and needs for 
use throughout the delivery system, particularly at the local and state levels. 

By these actions, Congress has established a general framework for repacking Pandora's box. 
The following section sets forth an analytical framework for understanding accountability 
and performance standards in vocational education. 



^This study is one part of NCRVE's response to that direction. 
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2. DIMENSIONS OF THE BOX: DEFINING ACCOUNTABILITY AND 

PERFORMANCE STANDARDS 



As a professional activity, education evaluation is very young, still grappling — passionately, 
at times — wiih basic issues of definition. For the pxirposes of this Report, three definitions 
are critical: 

• Accountability; 

• Performance measures; and 

• Performance standards. 

ACCOUNTABILITY 

Shorn of its technical language, accountability in vocational education is like accountability 
in any other public or private venture: It is a process to help people who expect specific bene- 
fits from a particular activity (and whose support for the effort is essential to its continua- 
tion) judge the degree to which the activity is working in their interests so that they might 
sustain it, modify it, or eliminate it. The preceding sentence describes the essence of the ac- 
countability process and, for the purposes of this Report, accountability really is that 
straightforward and that important. A more formal definition of accountability is drawn 
from a recent RAND Report: ^ 

Accountability is a relationship between two persons in which four conditions apply: first, one 
person expects the other to perform a service or accomplish a goal; second, the person performing 
the activity accepts the legitimacy of the other's expectation; third, the person performing the ac- 
tivity derives some benefits from the relationship; and fourth, the person for whom the activity is 
performed has some capacity to affect the other's benefits. 

An accoxmtability process is a continuing exchange of information and benefits, such that both 
persons are able to adjust their actions to create greater mutual advantage. The goal of the pro- 
cess is to improve the performance of the party responsible to deliver the service; but the process 
can also lead to termination of the relationship, if either person concludes that the expected bene- 
fits are not forthcoming. 

Under this definition, virtually any relationship between persons or legal entities (private 
contracts, public grants and contracts, and markets, for example) contains the seeds of an ac- 
countability process. 

But the most significant feature of the definition is that it establishes not a hierarchical pro- 
cess, under which senior government officials assess performance in light of their expecta- 
tions, but a much wider activity involving all the actors (public and private, funders and 
clients) who expect to derive benefits from the activity, and whose support or opposition can 
influence its future. The provider of a particular service is normally accountable not only to 
one person, but to several— perhaps to those who provide funding, establish the legal charter 



^v/WT^' Josephine J . Bonan. DfCfntralization and Accountability in Public Edtication (RAND, R-4066. 
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for the service, and benefit directly from the service (e.g., students, parents, or employers), as 
well as to the broader community, including its elected representatives. 

With respect to vocational education, many different people make judgments about its per- 
formance and possess the power to act on their conclusions. Members of Congress and state 
legislators, for example, may want vocational education programs to contribute to the eco- 
nomic development of their districts and may act on their hopes by changing statutory lan- 
guage, funding formulae, and budgets. Parents may care only that a particular program en- 
gages their children's interests or helps them find and hold a job. They can act by enrolling 
their children in one program or another, or by withdrawing them from programs. Local em- 
ployers may be interested primarily in whether the local schools are producing the air brake 
mechanics, cosmetologists, or electronics technicians reqxiired in their firms, and they can act 
by joining school vocational advisory groups and by supporting (or criticizing) school adminis- 
trators and public officials. 

Because each of the actors involved — Fe'ieral and state officials, state legislators, local edu- 
cators (including superintendents, program administrators, and school staff), parents, em- 
ployers, and students — comes to the table with different interests, an accountability process 
suited ideally for one set of "stakeholders** may be almost entirely irrelevant to the others. In 
fact, as will be shown in Sections 4 and 5 below, a process sxiited to the need of the most 
powerful stakeholder (ie., the federal government) may so distort the program that it fails to 
meet the interests of the least powerful, ie., parents, students, and local employers. 

For any stakeholder, the process of holding providers accountable involves both assessment 
and action. Performance is measured and compared to some standard, which is based on the 
expectations that both parties consider legitimate.^ Action (e.g., program change, reconfir- 
mation of commitments, rewards to providers, program termination) follows the assessment, 
and is dependent upon it. Performance data are not collected to satisfy idle curiosity, but to 
support effective change. As Levin noted: 

The perfectly accountable system is one that maximizes the utility or satisfaction of the relevant 
constituencies subject to resource (and possibly other) constraints. . . . Essentially, an account- 
ability system is a closed loop reflecting a chain of responses to perceived needs or demands; an 
activity or set of activities that emerges to fill those demands; outcomes that result from those ac- 
tivities; and feedback on outcomes to the sources of the demands. - . . The result is that an ac- 
countability system is continuous and dynamic when the linkages are tight and information is 
generated and transmitted freely. It is much less responsive when these conditions are not met.^ 



Types of Accountability 

The rhetoric of accountability often obscures crucial differences between hierarchical organi- 
zations, in which goals and methods are established clearly, and problem-solving organiza- 
tions, in which both goals and methods are fluid. In the former case (e.g., organizations 
manufacturing standard items) methods can be designed centrally and goals set in light of a 
clear understanding of methods. In the latter case (e.g., professional or public service organi- 



^An excellent source of definitions is Bruce L. R. Smith and D. C. Hague (eds.), Tfie Dilemma of Accountability in 
Modern Government: Independence Versus Control (New York: St. Martin's Press, 1971). 

^See Henry M. Levin, "A Conceptual Framework for Accountability in Education,** School Review, Vol. 82, No. 3 
(May 1974): pp. 363-392. 
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zations, which serve cHents whose needs frequently change) goals and methods are under 
constant revision. Moreover, the goals change both in light of chent needs and of the ex- 
pected effectiveness of methods. 

In truly hierarchical organizations, all that is required for accountability is a good manage- 
ment information system: Managers can monitor performance and intervene whenever it 
falls below set levels. Few organizations, however, are so fully automated that there is no 
need for initiative and problem solving by lower-level workers. In a problem-solving 
organization, however, accountability is much more complex: Inevitably, performance mea- 
surement involves judgments about what is needed and what is feasible under existing 
conditions. 

Recent analyses distinguish among four kinds of accountability: performance reporting, 
technical accountability, institutional accoimtability, and political accountability.^ The four 
differ on two dimensions: consensus on program goals and the process by which performance 
information is used. 

Levin's framework for accountability m education distinguishes between programs in which 
goals are clear and widely accepted and programs in which goals are diverse and open to ne- 
gotiation. 

As Levin argues, accoxintability issues are especially complex for programs whose goals are 
in dispute. In such situations, accountability processes inevitably are political: Actors who 
seek different goals want different information and draw different conclusions about what 
should be done to improve program performance. They want different performance informa- 
tion and put service providers under conflicting pressures. 

In vocational education, goals vary from place to place and from student to student. 
Moreover, there is some tension among program goals, as demonstrated by the fact that some 
local program operators emphasize job placement, while others concentrate on equity and ac- 
cess,^ Differences in goal orientation such as these can be illuminated, but not resolved, by 
performance information. Equity, access, and completion and job placement rates are politi- 
cal in their very nature, and no system of performance information can resolve the tensions 
among the values underlying each. 

PERFORMANCE MEASURES AND STANDARDS 

In situations that Levin would define as political, different actors emphasize different goals 
and make their own judgments about performance. Performance information is useful to 
them only if it reflects the goals they value. 

In the case of vocational education, interested parties can be divided into three groups: poli- 
cymakers, producers, and consumers. Policymakers are legislators and high-level adminis- 
trators at the Federal and state levels. They allocate funds and set the ground rules under 



*Ibid. 

^See Gareth E. Hoachlander, "Systems of Performance Standards and Accountability for Vocational Education: 
Gkiidelines for Development," (Berkeley, California: National Center for Research in Vocational Education, 1990); 
Norton W. Grubb and Lorraine McDonnell, Local Systems of Vocational Education and Job Training, Diversity, 
Independence, and Effectiveness, National Center for Research in Vocational Education, University of California. 
Berkeley, 1990. 
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which services are provided. Producers are the direct managers and suppliers of the services, 
and in this case would include school district administrators, heads of schools or postsec- 
ondary institutions, vocational education department heads, and teachers. For the purposes 
of this discussion, consumers can be thought of as two types, direct and indirect. Direct con- 
sumers are parents and students; indirect consvuners include both employers and other 
community members who benefit from the services provided. 

Working backward through this chain of interested parties, from the direct consumers to the 
policymakers, it is relatively easy to distinguish the different kinds of performance informa- 
tion each requires: 

• Students and parents need basic information about programs, e.g., the skills taught, costs, 
time to complete program, and employment prospects. 

• Employers need quite specific information about the skills of the people they are consider- 
ing hiring. While information about the program from which a particular student is grad- 
uating may serve as a reasonable proxy, the employer also is quite likely to be interested 
in the level of competency of the particular student. 

• Producers need information that helps them monitor aiii tailor their vocational offerings. 
Do graduates stay in this area or move elsewhere? What kinds of jobs exist in this com- 
munity now, and what kinds are likely to be created in the near future? What proportion 
of our students, in different programs, are performing at an acceptable range according to 
local employers? Which programs are no longer needed, and what new services should be 
supplied? 

• Federal and state policymakers need to know if services are provided equitably among dif- 
ferent states and districts and within districts to different population groups. They will 
want to know if the services provided appear to match community needs, as well as the 
level of satisfaction that local consumers, both direct and indirect, appear to experience 
with local programs. Generally, the policymakers will want to make sure that some out- 
reach and "sensing" mechanisms exist to determine local needs, and that local communi- 
ties have some way of assessing their own performances. 

The question remains, what are performance measures, and what standards can be applied 
to them? Hoachlander's guidelines in this respect are very useful.® He includes several 
criteria for performance measures: They should provide clear and precise definition; they 
should be qtiantifiable; they should be relatively easy and inexpensive to measure; they 
should be consistent across programs and institutions, timely, and consistent over time. 

What Are Performance Measures and Standards? 

A performance measure is a variable that tells us something about proficiency in accomplish- 
ing a specific task. A measure may be defined for students, courses, programs, institutions, 
districts, regions, or states. For example, a performance measure for students might be the 
time it takes to complete an overhaul of front disc brakes; for a program or institution, the 
number of students successfully completing an automotive program, or the number of stu- 



^See Hoachlander, op. cit. 
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dents completing a planned sequence of vocational and academic courses. For a district or 
state, a performance measure might be the number or proportion of students completing spe- 
cific programs who find employment, or who find emplo3rment in the area of their training. 

Performance standards, by contrast, are specific values on a performance measure that indi- 
cate minimum, adequate, and high levels of performance. Using the measures discussed 
previously, a minimum stuuent level of performance might be completing the front disk 
brake overhaul in two hours; adequate performance in 90 minutes; and superior performance 
in 45 minutes. For an institution, 90 percent of students completing the automotive program 
might represent superior performance; 80 percent, adequate performance; and 70 percent, 
minimally acceptable performance. And for a district, similar proportions of students finding 
emplo3rment might represent appropriate benchmarks. Additional standards readily can be 
imagined: For a student, acceptable performance can be demonstrating particular competen- 
cies, advancing to the next level of education or training, or obtaining a job related to train- 
ing. For an institution, acceptable performance can be maintaining full enrollment in 
courses, graduating a set proportion of students, or placing a percentage of graduates in jobs. 

An important function of a performance standard is to trigger remedial action. Congres- 
sional and state interest in performance standards rests on something more than a vague 
belief that they are good things to have. If a trustworthy system of performance measures 
and standards can be put in place, deficiencies can be illuminated, leading, ideally, to cor- 
rective action. 

Need for Multiple Measures 

Just as different actors have different information needs, so too different types of information 
must be available to serve those needs. Researchers appear to agree on the importance of 
three types of performarje measures: learning measures, student labor market outcome 
measures, and measures of community-wide support. 

Learning Measures. These measures concern the acquisition of academic and vocational 
skills. Student achievement on standardized academic tests and on competency-based tests; 
rates of course and program completion, degree or certificate attainment, and high-school 
graduation; and patterns of course taking (e.g., proportion of students pursuing a planned 
sequence of vocational and academic courses) all are examples of performance measures re- 
lated to learning goals. 

Labor Market Measures. These measures assess performance in the labor market after 
leaving vocational education programs. One obvious measure would be placement rates; an- 
other would be placement in work related specifically to the training provided. Other exam- 
ples include the time needed to find and secure employment, placement in further education 
and training, opportunities in the armed services, time employed in the first job, employee 
and employer satisfaction, and various wage indices such as entry -level wages, quarterly 
earnings, or rate of increase in earnings. 

Community-wide Measures. These outcomes are much more complicated to monitor. 
They might include such indicators as the economic opportunity structure in the area (i.e., 
skills and labor market outcomes for students who had been victims of discrimination); voca- 
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tional education responses to emerging business needs: and broader effects on the local econ- 
omy, such as the health of local firms or increases in aggregate activity. 

Part of the rationale for multiple performance measures is the ^'corruptibility'* of individual 
measures.'^ Individual performance measures or indicators are corruptible if it is possible to 
affect the indicator outcome without any real change in the underl3ing behavior it is in- 
tended to measure. Thus, for example, students' average grades are relatively easy to cor- 
rupt, since they can be raised through a lowering of standards as readily as by an increase in 
student performance. To guard against the possibilities of corruption, most analysts recom- 
mend that important outcomes be measured in several independent ways and be continu- 
ously refined. Senior policymakers may think they want one kind of information, such as 
whether students are learning or getting jobs, but without contextual information about the 
manner in which programs work and the problems that must be overcome, officials are likely 
to misunderstand reported outcomes and take actions that might make programs less effec- 
tive in the long run. 

VOCATIONAL PROGRAMS: SPECIAL PROBLEMS 

Based on this analysis, it is quite clear that accountability processes cannot be designed in 
the abstract. They must reflect the character of the services provided, the goals of the actors 
involved, and the abilities of providers to improve their own performance. The core argu- 
ment of this paper is that the realities of vocational education frustrate any effort to build a 
hierarchical accountability relationship between Federal officials (members of Congress and 
executive branch administrators) and local program operators. This is so for two reasons. 
First, Federal officials' go^ls for vocational education are diverse and conditional: They want 
local service providers to serve local community needs, which may differ dramatically from 
one place to another. Second, the effects of Federal funds and regulations are not traceable 
to the local level, because there are no imequivocal differences among the types of vocational 
education supported by local, state, federal, and private sources. The confounded nature of 
vocational education programs defeats the effort to measure the effects of specific federal ac- 
tions. 

Diversity of Groals and Services 

Vocational education is a general t^rm — not simply a broad umbrella, but an enormous tent 
sheltering a remarkable variety of very different activities. Vocational education is delivered 
in public high schools, commimity colleges, proprietary secondary and postsecondary institu- 
tions, and community centers. It covers a range of technical and practical subjects, from 
avionics to cooking, and can impart highly refined skills in mathematics and science, or very 
basic ones in arithmetic, reading, and simple record keeping. 

The goals of different vocational education programs vary from place to place and from stu- 
dent to student. Some vocational courses ars integral parts of a program sequence leading 
directly to a highly skilled occupation (e.g., technician for snrall shops specializing in com- 



'^See William W. Cooley, '^:ducational Indicators within School Districts," in Ron Haskins and Duncan MacRae 
(eds.), Policies for America's Public ScJiools: Teachers, Equity & Indicators (Norwood, New Jersey: Ablex Publishing 
Corporation, 1988). 



puter*assisted machining operations). Other courses (e.g., t)rping or homemaking) provide 
highly specific skills that may be as useful for university-bound students as for students who 
go directly to work in jobs requiring fewer skills. Because students in a single institution 
(e.g., a comprehensive high school) may take the same course for disparate reasons, there are 
no consistent relationships between the character of instruction provided a student and the 
intended outcomes of that instruction. Nor, for that matter, is there any necessary relation- 
ship between the kind of institution providing the instruction and the intended outcomes for 
the student 

In this environment, aggTegate measures that ignore the diversity of places and programs 
caimot gauge accurately the performsnce of vocational education. 

Ambiguity of Cause and Effect 

Given that diverse goals and services complicate efforts at aggregate measurement^ a second 
complication, possibly more compelling than the first, is that vocational programs receive 
support from many sources, making it impossible to identify any given program outcome 
with a specific funding source. Many different public and private entities fund and operate 
vocational education programs. There is no direct relationship between the identity of the 
funding source and the activities undertaken under the heading vocational education. Fed- 
eral, state, and local funds are intermingled freely in supporting each of the types of insti- 
tutions that deliver vocational education, and each type can cover nearly the complete range 
of sxibject matters and skill levels. 

The outcomes of vocational education are measurable at the level of the individual student, 
or as averages representing central tendencies for programs, institutions, or geographic ar- 
eas. However, it is very difficult to attribute any particular outcome, either at the student or 
institutional level, to the actions of one source of funding or authority. In this respect, the 
task of assigning outcomes to a specific funding source is immeasurably more complicated in 
vocational education than is the case in Chapter 1 of ESEA. Chapter 1 services and students 
are, by law, carefully kept separate from regular school services. But funding sources, ser- 
vices, and students in vocational education cannot be distinguished at the classroom level. 
Furthermore, when vocational outcome measures are aggregated, either across programs or 
across geographic areas, inevitably they confound the results of programs with different 
goals, funded by different sources. 

Effects of Other Educational Opportunities. The attribution of outcomes in vocational 
education is furttier confused by the fact that vocational education students receive instruc- 
tion from other sources, as well. Their intellectual skills and labor market outcomes cannot, 
therefore, be attributed solely to vocational education. Students learn to read, analyze, and 
compute throughout their school careers; their overall level of success and failure cannot, 
with integrity, be attributed solely to their experience in vocational education. They develop 
their understanding of what it is to work and how adults operate in organizations from ob- 
serving their own teachers in elementary and secondary schools. These lessons can be well 
taught both in vocational and other school programs; they can be poorly taught in schools in 
which adults take little responsibility for the results of their work.® 



®See Paul T Hill, Gail E. Foster, and Tamar Gendler, High School With Character (RAND. R.3944-RC, 1990), 
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Within a school, school system, or postsecondary institution, vocational education is part of a 
broad repertoire of programs. Managers of those institutions pursue the institution's overall 
goals, rather than the individual goals of outside funding sources. Though student assign- 
ment practices are not always enUghtened, it is clear that many school heads consciously al- 
locate students to programs to maximize student outcomes, not to maximize the apparent 
outcomes of particular programs. In such a case, the institution's goals can conflict with the 
goals of particular, externally funded programs: Students might be assigned to vocational 
education because they will do better in such programs than any others the institution has to 
offer. Those students, however, might uot be the ones who would create the appearance of 
greatest success for the vocational education programs. In such a case, the institution may 
have met its own goals extremely well, and the vocational education program will have made 
a ma^or contribution, but the vocational education program, in isolation, may not look highly 
successful. 

Efforts on the parts of higher authorities and external funding sources to gxiarantee clear 
identifiability of the services they fund can distort institutions' program offerings by requir- 
ing special pullout programs for beneficiaries, for example, or by encouraging managers to 
"cream" the student population in order to inflate apparent outcomes. In general, pressures 
to maximize the apparent effectiveness of programs that, like vocatioral education, operate 
as integral parts of broader institutions frequently have perverse results. 

CONCLUSIONS 

Based on this analysis, the following appear to be the dimensions of a box that can contain 
performance standards for vocational education: 

• Performance measures and standards are valuable only insofar as they inform judgments 
and focus action. 

• Performance standards must be understood as part of an accountability process under 
which all the actors who expect specific benefits (and whose support for the effort is essen- 
tial) can judge whether it is working in their interests and can make decisions about 
whether to sustain, modify, or eliminaLe it. 

• The larger accountability model should recognize that programs serve diverse purposes, 
and that performance information will be used in varying ways by different groups, each 
guarding its preferences and its stakes in vocational education programs. 

• Performance measures should be developed around at least three outcomes, learning, stu- 
dent success in labor markets, and commtmity-wide support for vocational programs. 

• Because the realities of vocational education frustrate any effort to build a hierarchical ac - 
countability relationship, performance standards should be designed to help local service 
providers serve local needs which may differ dramatically from place to place. 

These dimensions represent a very large agenda, one oriented toward encouraging accoiuit- 
ability for local performers to meet local needs. If that agenda is to be realized, the major 
onus will fall on states and localities. Immediately a question arises: Do they possess the 
capabilities to respond? As the next section makes clear, some states and localities do, while 
some do not. Most possess capabilities somewhere in between. 



3. STATES AND LOCALITIES 



As the Perkins amendments made their way through the legislative process to the 
President's desk, the economy was reaching the end of one of the longest peacetime expan- 
sions in American history. States and localities, flush with a decade-long expansion in tax 
revenues, were able to increase education ftmding dramatically in the 1980s. But as the 
Perkins amendments became the law of the land, the period of economic expansion ended, to 
be replaced with nationwide headlines about state and local budget deficits and their effects 
on non-entitlement spending: reductions in education funding; cutbacks in public assistance 
and aid for the homeless; layoffs and furloughs (forced, unpaid leave) for state and local gov- 
ernment employees, including educators; rising tuitions accompanied by layoffs at piilic 
colleges and imiversities; and shelved plans for program and building expansion. 

The Perkins amendments added to these difficulties for state education agencies. Prior to 
1990, the Federal set-aside for vocational education administration at the state level was 
7 percent of each state's basic grant. The 1990 legislation, intent on targeting the highest 
possible proportion of funds to the local level, reduced the set-aside for state administration 
to 5 percent of the basic grant. At the same time, the legislation clea-ly asked state 
education agencies to play a much more active role in developing performance standards. 
How do states and localities understand these new accoimtability requirements? How have 
they responded in the f ?<:ai environment in whidi they find themselves? How nearly do 
existing state and local accountability processes come to fulfilling the Perkins requirements? 
What are states and localities doing to fulfill the requirements? 

In order to explore these issues, state and local vocational education administrators in New 
York, Pennsylvania, Maryland, Ohio, Oklahoma, Indiana, and California were interviewed in 
late 1990 and early 1991. In addition, attempts to trace the effects of state actions were 
made via interviews in a small number of local education agencies. The interviews explored 
four broad questions: 

1. Current efforts. What is the State Department of Education doing now to hold local voca- 
tional education providers accoimtable for program quality and student outcomes? 

2. Client information. What information, if any, does the state now supply to local busi- 
nesses, civic leaders, and parents about the performance of vocational education pro- 
grams? 

3. Plans. What does the State Department of Education plan to do in response to the 
Perkins Act performance standards and accountability requirements? 

4. Trade-offs. What current state-level accountability or program improvement efforts are 
beiiig abandoned, in response either to the Perkins requirements or to state budeetarv 
limits? ^ 

The interviews elicited an amazing range of responses. Some states already were deeply en- 
gaged in an effort to "manage vocational education with data," keeping detailed accounts of 
program enrollments and outcomes, providing local administrators with information compar- 
ing their efforts with others across the stato, and pressing for the elimination of programs 
with ff>lhng enrollments or lagging job placement rates. Others were limited by a pervasive 
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state culture of localism, unable to imagine that they could take aggressive regulatory actions 
or put local providers under pressure by publicizing »esults. 

Some of the more aggressive respondents saw the Perkins requirements as opportunities to 
press reforms they had long favored. Testing advocates particularly were encouraged. As 
one said, *The requirement for gain scores can mean only one thing: fall and spring testing 
of every vocational education student." Another echoed. This is the one chance to get true 
statewide testing for all vocational education. Without these federal requirements it would 
be much tougher to overcome local resistance.* 

Budgetary crises and strong traditions of localism made other respondents much less enthu- 
siastic. As one state agency official said, *This state has been abandoning statewide tests 
and other expensive measurement as fast as it can. Nothing in the Perkins Act is going to 
change that." The vocational education director in another state said, "Every sort of educa- 
tional funding is being cut back. The state office is the first thing to get cut, but that hap- 
pened years ago. We are now cutting into the bone and muscle in the schools. This is no 
time for an expensive new testing program." 

Others embraced the principle of state-level responsibility to ensure that local programs were 
monitored closely and held accountable, but they argued that states mxist have flexibility in 
devising responses. One state vocational administrator said, *The law [Perkins] can only 
mean for us to do what is possible. We can try out some testing and think about what a 
statewide set of records might be like. But we can't do it for the whole state now." 

Another emphasized that states cannot fulfill all their responsibilities by measuring out- 
comes and publicizing results. As one state official responsible for program accreditation 
said, ^'It is good to have statewide indicators, the more varied the better. But they don't tell 
us how we should act toward any school or program. That takes a close look at the local con- 
ditions." 

Despite these differences, vocational education officials in all states were concerned about 
program quality and determined to improve it. The results of interviews in the seven states 
are summarized below, highlighting circumstances in those states that appear best and least 
able to respond to the Perkins requirements. Because the detailed information is now more 
than a year old, extensive summaries of the other states are not included here. 

Table 1 outlines the situation in the seven states with respect to each issue. The evidence 
from the states surveyed in this study reveals a remarkable array of ongoing and planned 
state activity. Many state and local vocational education programs were embedded in well- 
developed accountability systems, in which program quality assessment was valid and linked 
to plausible program component processes. Others have weak accountability processes. All 
such processes, strong or weak, are compromised severly by the financial condition of indi- 
vidual states and by state legislative support for education, generally, as well as for voca- 
tional education. 

I 

THE GOOD NEWS ... 

In some states, such as Indiana and Oklahoma, the state education agency has been able to 
maintain impressive, often elaborate, statewide accountability mechanisms in the face of 
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Table 1 
State Summary 



SUte 



Current Effort 



Client Information 



Plans 



Constraints 



California Annual survey 

Accreditation reviews 



None currently 

Report card will 
provide 



Indiana ISR (Indiana Student None currently 

Reporting) system of 
student data 

Maryland 13-point evaluation pro- None currently 

gram. Profile on each 
graduate. SEA on-site 
visits 

New York Still supports VEDS 



Ohio Still supports VEDS 



ConsTiltants examine 
poor programs 

Placement rates — 84% 
candidates for closing 

Oklahoma Elaborate accountability 

apparatus 

Continued VEDS; added 
new surveys of stu- 
dents, graduates, and 
employers; complex ac- 
creditation review pro- 
cess 



Pennsylvania Lost VEDS 



Surveys localities on 
graduates' placement 
and earnings after 1 
year 

5-year state review re- 
quirement a "paper exer- 
cise," except for reviews 
of9AVTSs 



None currently 



Data on economic 
trends and job 
demand 



Surveys of student 
outcomes 

Accreditation visits 

State mandate for local 
industry advisory 
bodies 

Economic and labor 
demand projections 

State-required local 
employer advisory 
group very effective 
in rural areas 

Keystone skills 
certificate testing 
program designed, 
but not in widespread 



Add vocational programs 

to report card 
Add academic content to 

vocational program 

Poasibly combine ISR 
system with VTECS 
to piDvide warranty of 
graduates 

Add vocational education 
to school report card 



Task force on standards. 
May go to school report 
carc^ system. 

Unresolved conflict over 
pre- and posttesting 



Strong compliance 
orientation 

170-plus detailed new 
curricula 

Competency tests for 
new curricula 



Task force c jnsidering 
Perkins requirements 

Possible report card 



SUte deficit 

No effect on 
Perkins 
standards 

None expected 



State deficit 



State deficit 



State technical 
assistance reduced 
dramaticaUy 



Centra] staff cut 



None 



State budget prob- 
lems have elimi- 
nated state ac- 
countability 
mechanisms 
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cutbacks in Federal support for state agencies and Federal abandonment of the Vocational 
Education Data System (VEDS). These states consider their efforts to be independent of, and 
often to precede, Federal interest in performance standards. State officials generally are 
confident that their own plans for accountability and performance improvement easily will 
satisfy any Federal requirements. 

Indiana and Oklahoma are states in which vocational education is run by a separate organi- 
zation that is not subordinate to the state education agency. This means that the leaders of 
vocational education have direct access to the state legislature and can deal independently 
with local education agencies. Independent state vocational education agencies can develop 
their own strong relationships with imiversities that train teachers and develop curriculum, 
as well as with community colleges that deliver or supplement vocational education services. 

In contrast, state vocational education offices that are subordinate parts of the state educa- 
tion agency must coordinate legislative advocacy and local oversight with other elementary 
and secondary education programs. At the state level they suffer from the same cuts in staff 
and funding as affect other education programs. In many states they also are imder pressure 
from the governor and business community to adopt an assistance-oriented mission, aban- 
doning the federal programs administration and compliance monitoring functions that have 
dominated state agency missions since the early 1970s. In Ohio this pressure has taken the 
form of a major report by the business commimity condemning the emphasis by the State 
Department of Education on administration of federal programs and demanding adoption in- 
stead of a totally different assistance-oriented mission (Ohio, 1991). 

Oklahoma 

Of all the states visited, Oklahoma has by far the largest and most elaborate state vocational 
education administrative apparatus. The state has more people working on curriculum de- 
velopment, alone, than any of the other states have for their entire vocational education of- 
fice staff. The state's vocational education accountability apparatus, already complex, 
rapidly is becoming more so. Having had no difficulty continuing the VEDS system, the state 
has elaborated that system with a number of its own statewide surveys of students, 
graduates, and employers. It has an extensive statewide accreditation system, as well, under 
which teams of experts (including state officials, employers, academics, and professionals 
from other local vocational education programs) visit local programs for a week to review 
program quality. 

Though all of these activities predate the Perkins requirements, state officials have no plans 
to abandon them. What is new since Perkins is a system of competency tests being developed 
for each of the more than 170 separate vocational education curricula that the state office has 
developed. The new curricula prescribe in detail the course content and instructional se- 
quences to be followed in specific occupational areas (e.g., structural carpentry, concrete 
forms carpentry, and cabinetry). The state vocational education agency is developing compo- 
nent skills lists (based on industry advice) for each of these 170 areas. Then it plans to build 
a complete battery of competency tests for each group of skills lists. The tests include ques- 
tions of fact, written scenarios testing inference and judgment, and live performance of criti- 
cal tasks. The tests are being developed and calibrated by administering them to skilled 
journeymen in the occupations under assessment. The state then will establish a cutoff score 
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for successful completion of the tests; for example, a student must answer correctly 75% of all 
the items on a given test in order to be ce.tified as competent. 

These tests are under development still, and complete tests exist now only for about ten oc- 
cupations; however, the task lists that will be used to generate test items are almost complete 
for all currinila. Ultimately, according to state officials, this testing program will be the 
main medium of accountability. The state will use the program to certify programs. Local 
education agencies will use it to determine whether to start or terminate programs, and 
employers will use it to make hiring decisions. 

These plans for testing complement the existing accoimtability structure of student surveys 
and periodic accreditation visits. Also, the state requires localities to maintain industry ad- 
visory committees for every m^gor program. Given the high level of staffing at the state office, 
the surveys and accreditation visits are particularly effective management tools: The state 
supports consultants who look closely at programs with poor results, and several (fewer than 
ten) are closed every year. 

Because most vocational education programs are in rural areas and small towns, local advi- 
sory groups truly can be representative of local employers who are, therefore, highly influen- 
tial. Some local Area Vocational Technical Schools (AVTSs) also are experimenting with 
skills certificates attached to a student's high-school diploma; these certificates describe in 
some detail the skills a strident has mastered. 

Oklahoma offers an impressive array of information services, mostly used by local vocational 
education programs, not by the clients of the local programs. The state supplies basic tabu- 
lations of its VEDS-like data to all local programs and maintains a hot line for special analy- 
ses (e.g.. How does the placement rate for my auto mechanics program compare to the 
statewide average?). Most hot line questions come from local providers, not from employer or 
parent groups. In general, except for large cities, the employer advisory groups are influential 
because their members truly represent local employment opportunities for graduates and 
imderstand local job needs. 

The state has no intention of abandoning any of these efforts. Although the state director 
complains about reductions in federal support for state-level operations, his access to the 
legislature has, to date, guaranteed ample fimding. The costs of the testing program have 
not yet been fully realized. But now, with only a few tests completely developed and the en- 
tire scheme being pilot-tested in three schools, there is no talk about costs or burden as an 
impediment to the state's plans for performance standards. 

. . . AND TEE BAD 

In other states the combination of Federal changes and statewide budget calamities have 
severely imdercut state agency efforts to improve program performance. Both Ohio and 
Pennsylvania are struggling with the effects of budget shortfalls on statewide administration 
of vocational education. 



22 



Pennsylvania 

Vocational education is healthy and vibrant in many Pennsylvania localities, but at the state 
level it is in bad shape. In some ways, Pennsylvania is the kind of state for which the new 
Perkins requirements were written — a state that has no real accountability or program im- 
provement structure for local programs. Although the requirements have motivated state of- 
ficials to think about what they will do, they have few assets. It might be a long time before 
they are able to create any real statewide effort. The state has been unable to maintain 
VEDS, and although the Governor and the Secretary of Education speak enthusiastically of 
requiring school report cards (which may or may not involve vocational programs), the truth 
is that state budget problems already have led to reduction in state staffing and funding for 
travel and consultants. The state's capacity to measure local performance, or to act on per- 
formance information once it is obtained, is very low. For example, state-mandated five-year 
reviews of vocational programs have, in the eyes of state administrators, been reduced to Ht- 
tie more than paper exercises. 

Like most other states, Pennsylvania assembled a task force in 1990 to consider ways to re- 
spond to the performance standards/measures requirements in Perkins. The task force had a 
daunting assignment, however, because the state had dropped its VEDs system in the early 
1980s, as soon as the federal requirement was lifled. In general, according to the 
Pennsylvania vocational education director and senior staff, the state is strongly oriented to- 
ward local control. During recent state budget crises, that principle was used to rationalize 
dramatic reductions in state staff and activity. Although the state office still surveys local 
vocational programs about experience of graduates (i.e., placement and earnings) one year 
after completion, those data have not been used regularly to evaluate or to manage pro- 
grams. 

The task force has consulted with similar groups in Ohio and California, and its members are 
considering a possible competency testing system. There is serious doubt, however, that 
funding for such a system ever will be available. The legislature forced cancellation of two 
prior testing programs (due to budgetary considerations and pressure from local education 
agencies), and now the state has no statewide testing program. The governor is in favor of a 
report card system like California's or Maryland's, and the Secretary of Education strongly 
supports the concept. But if such a system is created, it will not address vocational education 
immediately, and it remains unclear whether there will be sufficient funds to include voca- 
tional programs at any time in the next several years. 

The state office's job is complicated by the structure of vocational education in Pennsylvania. 
The office deals directly with 87 technical schools, 9 area vocational-technical schools 
(AVTSs), 87 vocational schools, 13 commimity colleges, and 501 local education agencies. In 
theory, every program, including every vocational education institution and comprehensive 
school offering vocational education, has a review every five years. The review should in- 
clude an application, self-assessment, site visit, and feedback. Given the size of the state 
staff (fewer than twenty), however, the schedule is unrealistic. State staff members visit a 
few of the 1400 programs each year, but the five-year review has become, at best, a paper 
compliance effort headquartered in Harrisburg. The state provides economic and labor de- 
mand projections of a highly general sort, but it does not provide program comparisons or 
any standard outcome measures. That would be the purpose of the governor's report card 
scheme. 
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The state requires each vocational education prograiQ to have a local employers' advisory 
group, and these are active, especially in rural areas and for specialized vocational and tech- 
nical schools. Employers' advisory groups suggest needs for new courses, advise on trade-oflFs 
between existing courses and potential new ones, and help develop curricula. Unlike 
Oklahoma, with its highly centralized system, curriculum development in Pennsylvania is a 
local activity. Such advisory groups are less active and influential for programs in compre- 
hensive high schools, especially in the big cities. The state office has pushed the idea of com- 
petency certificates and has provided training materials on ways local programs can incorpo- 
rate such certificates into diplomas and call employers' attention to them. The state office 
also has created a program called the Keystone Skills Certificate. This is a rigorous testing 
program designed to demonstrate students' attainment of very high degrees of technical 
skills in several fields, most related to electronics. 

Testing in the Keystone Skills program is extremely demanding. Only students with ad- 
vanced reading and thinking capabilities, as well as technical skills, can pass it. Although 
the state has paid for the design work, the local programs must pay for the tests, which are 
administered individually and take nearly foiir hours to complete. The Keystone program 
has not caught on, really. Only a few local programs are using it, and the state has no means 
of determining whether the certificate is helping graduates get jobs. Some state officials 
worry that the demandingly high standards can be met only by those students so capable, 
they will go straight to college or will be hired by employers immediately, with or without a 
certificate. 

At the state level, Pennsylvania seems to have nothing left to lose with regard to the new 
Perkins requirements. Vocational education programs may be healthy at the local level, be- 
cause of the structure of the Pennsylvania economy and the close-knit nature of the smaller 
communities, but the state no longer is able to promote improvements in local program qual- 
ity. The governor's report card scheme is the only state-level performance measurement ac- 
tivity on the horizon, and it will not aflTect vocational programs for a long time, if ever. 

MISED REVIEWS 

Most states appear to have neither the ambition and self-confidence of Oklahoma, nor the 
problems of Pennsylvania. They have some successes to report in the area of performance 
standards, as well as some failures. Some states have been able to maintain programs in the 
face of state and federal reductions, but were unable to mount new efforts. Others have sal- 
vaged some efforts, but have had to abandon others. Most other states visited had a far 
healthier system of state monitoring and accountability than Pennsylvania has been able to 
maintain, but had far fewer varied and ambitious systems than Oklahoma boasts. All but 
California had continued a VEDS-like program data system. Indiana also was developing a 
statewide system of computerized student records that could be used to track enrollments, 
student progress, and initial labor market success. Most had sustained the practice of peri- 
odic local self-analysis leading to external accreditation reviews. These reviews took place 
once every three years in Ohio and every five years in California; over time, state staff mem- 
bers have assimied a smaller role in these processes, as cutbacks depleted manpower. 

Many state directors expected vocational education to be included in broader statewide test- 
ing programs and school report card systems. California and Maryland, for example, ex- 
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pected new statewide school report card systems eventually to include indicators of voca- 
tional education enrollment, completion of industry-focused training, and integration of aca- 
demic and vocational training. 

Vocational education directors and evaluation specialists in several states hoped to create 
competency testing systems like Oklahoma's, but none were confident that funds to develop 
such a system could be found. Some state officials, thinking that the Perkins Act language 
on student competency gains required fall and spring testing of all students, despaired of 
coming into compliance in time. Others understood that Congress had not intended to man- 
date a whole new testing system for any state. In general, most states maintained a combi- 
nation of formal measurement systems and informal site visits to assess program quality, 
identify areas in need of improvement, and target assistance. The majority had experienced 
cutbacks in state agency funding, and thus were making do with fewer staff members, fewer 
consultants, and less money for travel. Some saw the movement toward testing as a way to 
substitute technology for people. But most senior state officials thought they had less infor- 
mation about local programs than in the past, and they could do less to help those in need. 

VIEW FROM TBDE LOCAL LEVEL 

In addition to the state level interviews, informal local site visits were conducted in 
Pennsylvania, Ohio, Oklahoma, California, and Indiana. In addition, school-level cases writ- 
ten for other RAND and NCRVE studies were analyzed in Ohio, New York, Virginia, Florida, 
Kentucky, California, and Oregon. The resulting data base covers rural, urban, and 
suburban areas, as well as specialized vocational education institutions and comprehensive 
high schools that provide some vocational programs. The goal was to understand the ways in 
which local programs are held accountable to employers, parents, students, and the local 
public education authorities. 

Though data were collected at the LEA, school, and program levels, complete case studies of 
the larger and more complex systems could not be conducted. The result is a national sample 
of local programs and their accoimtability processes, rather than complete case studies of 
particular localities. 

The overwhelming sense derived from local interviews was that in small, particularly rural, 
communities, accountability processes work reasonably well, but that they are much more 
problematic in larger, urban, areas. In very small communities with only a few employers, it 
is possible for healthy accountability processes to work in everyone's interests. As the head 
of a small-town vocational center said, "All the businessmen know one another and know the 
families of virtually all the kids going through the schools. It is easy for the business people 
to know whether the school programs are producing the workers they need, and it is easy for 
parents to know whether their kids are being prepared." 

In a small town or rural area, the range of employment opportunities is finite and generally 
understandable, and almost all employers who care to can participate in the normal pro- 
cesses of consultation. Vocational education teachers know their communities, often taking 
the initiative to seek out employers to assess the quality and relevance of the teaching they 
provide. Vocational education administrators and teachers serve as placement agents both 
for students and employers. Performance feedback is quick and direct: Providers cannot 
hide behind bureaucratic routine-following. In larger communities, however, only a fraction 
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of employers can be consulted directly. Vocational education directors, like one in an Indiana 
city, must *^ork through formal meetings or advisory boards. No one could have direct con- 
tact with even 1% of the employers in an urban labor market." In big cities, larger and 
longer-established employers are easier to identify and to engage in collaboration. They are 
far more Kkely to be consulted than small or newer employers. In some places this means 
that important employer interests — ^the small firms that are the normal employers of new 
high-school graduates and the ne^ver firms that may rely on unfamiliar technologies and of- 
fer unconventional career opportunities — ^may be poorly represented. 

As vocational education administrators in several m^or urban areas told us, the work of 
teachers and administrators also is affected by their being parts of large, formally organized 
systems. They are assigned and evaluated according to formal procedures designed to limit 
individual discretion and to prevent "undue** outside influence in professional activities. The 
result is that providers often are insulated firom employers and get little direct feedback 
about graduates' work experience. 

The best local accountability processes ensure the quality and relevance of vocational educa- 
tion programs through a variety of mechanisms that rely, in the main, on the relevance of vo- 
cational program offerings to local job demand. To the extent that performance standards ex- 
ist, they revolve around placement rates and employer satisfaction, not around proportions of 
students in various skill areas able to complete specific tasks. Mechanisms such as the fol- 
lowing are common: 

♦ Employer advisory committees are established, frequently to assist discrete programs, but 
often to provide advice on emerging needs virtually in every area of vocational offerings, as 
well as to school- and district-wide planning for vocational education. 

♦ Some localities track employer needs routinely in their student placement efforts, as well 
as through follow-up surveys of graduates and of employers. 

♦ Many vocational programs make their success placing graduates in jobs related to their 
training — a recruiting device to interest prospective new students and their parents. 

♦ Some LEAs and postsecondary programs actively seek employer approval and certification 
of potential new course offerings, initiating, eliminating, or expanding courses based on 
student and employer demand. 

♦ Schools that* provide only vocational education depend on business sponsors for donations 
and support on local budget struggles and on the voluntary enrollment decisions of stu- 
dents and their parents. They often are quick to change programs that do not meet the 
needs of key local audiences. 

♦ Some metropolitan area vocational education providers arrange cooperative work experi- 
ences for students and send teachers to work for key employers in the summer. This cre- 
ates personal linkages even in areas too large to depend on routine personal contacts and 
friendships. 

The negative side of this picture, however, is that many local processes, particularly, but net 
exclusively, in urban areas, have serious weaknesses. The weaknesses are due not so much 
to issues of lack of competency testing as they are to the almost complete lack of effort to un- 
derstand local employers' needs or to adapt existing programs to job demand. For example: 
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* Tracking of student placement in many large commimities at best is lax, and at worst is 
nonexistent. Urban area teachers and administrators who keep track of their vocational 
graduates are the exception. Those urban providers who try to follow students do so via 
formal surveys. Given urban students* high mobility, however, the surveys provide little 
useftil information. 

* Many localities, both large and small, lack the capabilities to project emerging labor force 
needs, nor do they have access to state or r^onal data to help in thinking about program 
implications of these needs. Again, providers in urban areas have especially severe prob- 
lems, since the labor markets they serve often are much larger than the geographic area 
from which they draw students. Therefore, they cannot rely on their immediate local con- 
tacts for complete information. 

* Attention to student needs in some commimities is largely directed at the requirements of 
current students. Very little attention is given to the needs of transient students (e.g., 
children of migrant workers or of single parents who move frequently), or to the special 
needs of low-income mmority students. 

In such circumstances, vocational education teachers and administrators have only a dim 
appreciation of the economy they serve and the labor market into which they send students. 
As state administrators and local school superintendents in every state said, local providers 
easily can adopt the mentality of following an established routine, processing students 
through the systems and services already in place, and holding out very low expectations 
that students will attain skills or find jobs. 

POTENTIAL CONFLICTS 

A more subtle problem at the local level, even in those commxmities with ambitious efforts to 
stay in touch with employer needs, hes in the fact that the interests of students and parents 
easily can conflict with the interests of employers. As one state official explained, a well-es- 
tablished local employer may want students prepared for traditional low- and moderate- 
skilled jobs. A vocational education institution inclined to respond to the employer's needs 
may create programs that do not provide students with the skills they need to obtain em- 
ployment with other employers, possibly in other localities. An additional possible conflict of 
interest may arise between students and educators. As Cakes (1986) has shown, some school 
systems emphasize college preparatory education for all but the lowest-achieving students, 
and others regard vocational education as the only appropriate placement for low-income 
minority students. In those cases, vocational education can gain a reputation for being, in 
Cakes' words, a "dumping ground" for students with poor academic preparation. In this 
situation, educators adopt low expectations for students and offer them imchallenging 
coursework that does not prepare them for rewarding careers. As McDonnell and Hill (1993) 
show, a similar sequence of events appears to be developing for immigrant students who 
come to the United States either unschooled, or years behind their U.S. peers. Cider immi- 
grant students also tend to be assigned to less challenging courses that prepare them only for 
low-skilled dead-end jobs. 
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CONCLUSIONS 

Several impediments appear to block effective state-local cooperation in developing systems 
of performance measurement. One serious problem in several states is the lack of resources 
and personnel to support an effective performance measurement development effort. In this 
regard, the Federal government really is not entitled to have it both ways. It cannot reduce 
existing support for state administration, then, at the same time, require states to develop 
and administer a coirolex new accountability process, A second problem is insuflRcient at- 
tention to program quali^ and relevance in some LEAs, As the next section will show, this 
problem was a prime concern of members of Congress who initiated the Perkins Act perfor- 
mance standard requirements. Moreover, the possibilities for conflicts of interest among 
students, educators, and employers deserve closer attention in the development of perfor- 
mance measures. 



4. VIEW FROM WASHD^GTON 



From the preceding section it is clear that states and localities differ widely in the quality of 
provisions made for accountability and performance improvement. State officials also differ 
in their views of what Perkins requires and the extent of the challenge its requirements pose. 
At one extreme are those who believe the Perkins amendments require states to build and 
maintain new and elaborate centrally administered performance-measurement systems that 
will be the basis for holding local programs accoimtable and, possibly, for funding decisions, 
^ome truly embrace this view and are working hard to make it a reality. Others reject this 
^ lew as unworkable under any circumstances and impractical under current budgetary 
constraints. At the other end of the spectrum are state and local officials who believe that 
the intent of the Perkins legislation was simply to encourage states to strengthen local ac- 
countability processes. 

W>.at, ii. fact, did those most closely associated with developing the performance standards 
p^W;i8it;r.^ have in mind? The answer to that question was sought from Federal officials, 
hcth in Congress and in the U.S. Department of Education. 

These interviews indicate that Federal officials were not trying to find a formula for deter- 
mining effectiveness; rather, they were seeking levers to influence state and local activity. 
Senior Federal leaders lack the time, resources, or interest to monitor results. How, then, 
can they create a situation in which they c£Ui encourage program improvement without being 
required to make decisions about who will be rewarded or punished? The establishment of 
requirements for performance standards promises to solve this dilemma: Specifying the need 
for performance measures and standards forces implementers to live with the possibility of 
being closely observed and sanctioned. 

Virtually all of these officials emphasize that federal money is a very small part of total 
spending on vocational education. *The awful truth of vocational evaluation," said one, **is 
that the reputation of the effectiveness of Federal expenditures cannot be separated from the 
effectiveness of vocational education as a whole. For that matter, vocational education's 
performance is tied to the overall performance of the entire educational system." 

Under these circumstances, the Federal government has to rely on a carrot-and-stick ap- 
proach to modify the operations of well-established state and local programs. The carrot of 
Federal money, for example, can encourage local programs to take the needs of special popu- 
lations more seriously. The stick of disapproval of state applications establishes pressure to 
improve local performance. But Federal and state officials generally agree that although the 
Federal government talks loudly, the big stick it carries is not likely to be used. Withdrawal 
of support would, in fact, violate the existing statute, since, as one Congressional aide points 
out, the funds are awarded by formula— not on the basis of judgments about quality. Also, it 
would have the perverse effect of reducing support for programs in areas where students al- 
ready are suffering from poor program quality. 

Federal officials are acutely aware of the problem of the ambiguity of cause and effect in vo- 
cational education, as discussed in Section 2. Like state and local officials, Federal officials 
understand the operations of vocational education. At every turn they took the opportunity 
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to emphasize that there is no single national vocational education program, and that the vast 
preponderance of the money in vocational education is state and local in origin. The federal 
money is a marginal addition to a very large enterprise. 

THE FEDERAL INTEREST 

Despite their understanding of the modest role of Federal funding for vocational education. 
Federal officials have a substantial interest in using these fiinds to improve local programs. 
Congressional interest is rooted in several factors, according to the interviews: efficient use 
of public funds, legislative politics and the need to generate political support, concerns about 
the treatment of minorities and other special populations in vocational education, and the 
need to root out complacency in some local vocational education programs. 

Efficient Use of Public Funds 

The first factor in Congressional interest is found in the general accountability rhetoric sur- 
rounding education. As one seiiior Congressional staff member said, "In the early days of 
federal aid [to education] we took care of accountability through tight monitoring of spending 
and service delivery. Now, with the movement toward deregulation, the recipients of federal 
funds are accoimtable for results." Today, many business leaders are active in education and 
believe in accoimtability. As another aide said, "A solid-looking accountability scheme can 
generate business support . . . business is willing to support a bill that seems to ensure that 
the government will get a good deal." 

According virtually to all the Federal officials interviewed, accountability is, in many ways, 
little more than a buzzword: Because public entities are receiving public money, they must 
be answerable for its proper use, even if the range of appropriate uses is very broad. In the 
early days of Federal aid, these officials point out, accountability was accomplished through 
tight regulation and close monitoring of spending and service delivery. Detailed process con- 
trols caused complaints and often affected program quality adversely. Officials now hope 
that recipients of federal funds can be held accoimtable through "performance standards." 

The imposition of performance standards, however, goes beyond merely following the fash- 
ionable fad of the day. The new performance standards requirements in Perkins — in combi- 
nation with studies from the General Accovmting Office, the Office of Technology Assessment, 
and the National Assessment of Vocational Education — are meant to keep the pressure on, to 
insure that funds are used properly, and to improve the delivery of services to target popula- 
tions. As a senior U.S. Department of Education staff member said, "Congress likes to keep 
the pressure on. [Congress] listens to anecdotes and complaints and gets worried that funds 
aren't being used well. They commission studies and assessments to keep the states under 
pressure." 

Need to Generate Political Support 

Because performance standards promise greater accountability, they also generate support 
for vocational education from many constituencies that might be inclined otherwise to oppose 
Federal expenditures on edvication. Performance standards gained support for Senators 
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Edward Kennedy and Dan Quayle as they advanced the Jobs Training Partnership Act 
(JTPA). Similarly, legislation in the area of employability training might also generate sup- 
port as it promises greater accountability. 

A solid-locking accountabiUty scheme can generate support in tlxe business community and 
among the fiscally conservative parts of Congress and the Office of Management and Budget 
(0M3). One aide pointed out that a jobs bili with peifcrmanr3 standards prompted a letter 
of support from the U.S. Chamber of Commerce — a response that never had happened before 
regarding a piece of jobs-related l^slation. The Perkins accountability requirements bring 
vocational education up-to-date with other training programs such ^ s JTPA, whose require- 
ments have attracted political support. As one Congressional aide said, ^At the Federal level, 
vocational education has become a backwater, and ftiBcJing has stagnated relative to other 
education programs.*' At the local level, a similar dynamic seems to be at work: Vo^;aticnal 
education generally is left out of school reform eflForts that focus on raising test scores in tra- 
ditional academic areas and increasing college attendance rates. A realistic Federal cons- 
mitment to goal setting and accountability may help Federal support for vocational programs 
to survive and grow. 

Federal officials view business leaders as quite sophisticated consumers of perfoimance in- 
formation. "We idealize the business world, assuming that business people are closely evalu- 
ated according to results — profitability — and that consequences are swift and sure,** said one 
Congressional aide. "But'* he continued^ "that is probably net entirely tl*e case: Businesses 
have different expectations for different enterprises, even for similar operations in different 
places." Under this view, a requirement encouraging an open aiid ambitious process for set- 
ting local perform.mce standards is iikely to be perfectly acceptable to the business commu- 
nity. 

Concerns for Minority Students 

Despite the general popularity of accountability and performsr^ce standards, they are accom- 
panied by some imwanted baggage, particularly the possibility that performance standards 
might damage programs serving minorities and other special needs populations. 

Minority members of the House Committee on Education and Labor, including former chair- 
man Augustus Hawkins of California, believe that JTPA's performance standards encouraged 
programs to "cream,** that is, to choose *hose students who were most iikely to get good jobs 
even before the training. In many instances, creaming led directly to discrimination against 
blacks and Hispanics. In addition, some majority members of the committee, including 
current chairman William Ford of Michigan, are suspicious of schemes that might punish 
those programs whose students have the worst problems. 

This means, in effect, that as the performance standards requirements are implemented, vo- 
cational education administrators need to pay particular attention to the effectiveness of lo- 
cal programs in providing services to special populations. Surely, it is no accident that the 
accountability, evaluation, and performance standards provisions of the Perkins legislation 
(see Section 1) turn repeatedly to the issue of access of special populations to high-quahty vo- 
cational education programs. State plans must consider this issue, the National Assessment 
of Vocational Education is to examine it, and the National Center for Research on Vocational 
Education is to conduct research on it. Finally, among the Perkins Act provisions for 
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perfonnance standards, states aro required to consider incentives to encourage services to 
special populations and to assets iocal programs, in pr^rt, on their ability to meet the needs of 
special populations. For their part, localities are to develop their performance measures in 
consultation with students with special needs and the parents or representatives of such 
students. 

Attacking Complacency 

At the Federal level, few supporters of vocational education are more staunch than the 
Congressional stsff and Department of Education officials responsible for the Perkins Act 
As a senior House staffer said, ^Vocational programs are closely connected to local buf^ness 
needs, and everybody knows which programs are preparing effective workers for local jobs." 
Also, Congressional respondents noted that the situation is much more complicated in urban 
areas where the connections between schools i'oid local business are less direct. As one re- 
spondent noted, in some cities vocational programs are large and bureaucratic. Students are 
not provided with the services so much as they are processed through a system, and many 
programs continue long after private sector demand for their skills has disappeared. 

It is the view of these Federal officials that national and state leaders can see the effects of 
their investments in rural areas, but money disappears without a trace in many urban areas 
where too many programs are little more than hoJding patterns for students uninterested in 
any other aspect of high school. In these situations, nobody (students, teachers, parents, or 
administrators) really expects the students to learn enough to work in the area of their train- 
ing. This state cf affairs makes Congressional supporters of vocational education, particu- 
larly those from urban areas, defensive. Representatives from urban areas are not interested 
in embarrassing or negative information, but hope that the performance standards require- 
ments can "light a match" under some fairly complacent, entrenched local administrators 
who never have had to defend the performance of their programs. 

Perfonnance Standards Vereus Aggregated Data 

The Congressional solution to the tension between integrated local programs and separate 
federal acco^ontability was, in the words of a senior education committee staff member, '^to 
require performance standards, but to separate them from funding decisions." Once stan- 
dards are established, states have the responsibility to monitor the performance of local pro- 
grams and to offer assistance to those doing poorly. There is a vague long-term possibility of 
adverse funding consequences, but the consequences are not spelled out. 

Congressional supporters of perfcrmauco standards believe that most vocational educators 
will have little trouble wiA the accountability requirements as they have been defined. As 
one respondent said, "Vocational educators are accustomed to having to demonstrate 
whether their students get work and to working with local employers who frequently have 
strong opinions about whether students arc properly trained." Furthermore, the supporters 
argue that, compared to regular elementary and secondary education, the outcome measures 
for vocational education are more realistic and fair: Indeed, students' performance in the la- 
bor market is a better basis for accountability thj;n scores on standardized tests. 
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Why, then, does the Federal government worry about performance standards if the states 
themselves care enough to spend a great deal of money on vocational education? In voca- 
tional education, as in other areas of federal aid to education, the federal government some- 
times seeks to empower state and local officials who want to run their programs more rigor- 
ously, but need help to overcome entrenched provider groups. It is clear that Federal officials 
believe the standards keep performance pressure on the states. The thinking seems to be 
that it cannot do any harm to make the states expect scrutiny, to lead them through a search 
for appropriate standards and measures, and to make them contemplate the consequences of 
failure. Most states, of course, have some form of performance standards. But as a 
Congressional respondent said, "The Federal emphasis on ^standards and accoimtability' put 
the providers under new pressure to perform,* regardless of whether top polic3rmakers intend 
truly to mete out sanctions for poor performance. 

It is clear from the RAND interviews that the Department of Education's preference is to 
leave discretion at the lowest possible level. Officials believe that performance standards can 
help enhance local accountability if states assume the responsibility to measure and report 
on effectiveness and to increase program quality. The federal obligation, imder this ap- 
proach, becomes one of keeping the issue before state leaders, many of whom have not, in the 
past, been consistently rigorous about enforcing standards. 

PRACTICAL EFFECTS OF TBDE STANDARDS 

How, as a practical matter, do Federal officials see these new performance requirements 
working? Given the great diversity of vocational education programs and the strong political 
motivation for providing grants to all localities, it is difficult to make a case for removing a 
program's funds based on the performance standards. In fact, as is clear from the comments 
of Congressional and U.S. Education Department respondents in the previous discussion, 
most Federal officials do not foresee such an outcome, at least in the short run. As one offi- 
cial said, "Congress did not want to be trapped into removing funds from localities that des- 
perately need help just because their current programs are not as good as they should be." In 
the event of demonstrably poor performance, the Perkins provisions call for progressively 
more aggressive state technical assistance and program redesign. 

Federal officials envision a fairly lengthy and complex process for implementation of stan- 
dards. Once standards are set at the state level (a process that might take two or more years 
and will be complicated by the diversity of vocational programs), the legislation, in general, 
requires the following: 

♦ Recipient local programs and institutions, e.g., local education agencies, area vocational- 
technical schools, and community colleges, conduct their own self-evaluations in light of 
the standards. 

♦ Failure to meet the standards requires a locally developed plan to meet them. 

♦ After two years of reported failure, the state education agency will work jointly with ikie 
recipient institution, teachers, and parents. The state will provide technical assistance 
and will facilitate planning to create improvement, with a timetable for re-assessment of 
progress. 
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« The state-managed technical assistance and planning process will continue in subsequent 
years until the performance standards are attained for more than one year. 

Federal observers do not believe that these requirements impose an overly burdensome pro- 
cess on local recipient institutions. Most of the initial burden, they point out, will be on the 
states, at the front end, as they grapple with the difficult challenge of developing standards 
and assessment methods. States inevitably will have to face the problems of perverse incen- 
tives in any measurement scheme, ensuring both that programs show success and that they 
don't "cream* the eligible population in order to guarantee that they will look good. Although 
these considerations certainly are good news for local providers, they hardly can be encourag- 
ing to state agencies sufiTering through major staff reductions and coping with a general de- 
cline in the funding and capability of the entire state education agency. 

CONCLUSIONS 

Driven by four concerns grounded in legislative politics — a desire to ensure the efficient use 
of public funds, the need to generate political support, concerns about the treatment of mi- 
norities and other populations in vocational education, and apparent complacency in some lo- 
cal vocational education programs — Federal officials have imposed on states a requirement 
that they develop performance standards and measures against which the effectiveness of lo- 
cal programs will be assessed. 

Based on the judgments of Federal officials responsible for vocational education, it appears 
clear that the two worst fears of some state and local administrators about these new re- 
quirements are groimdless. The first fear was that the standards would be used to make 
fimding decisions. But Federal officials indicate that 'iie standards will be kept entirely sep- 
arate from decisions about eligibility for funds and level of funding, which are determined by 
statutory formula, not discretion. Once the standards are established — a process that may 
well take several years — states will have the responsibility to monitor the performance of lo- 
cal programs and to oflFer assistance to those experiencing the greatest difficulty. There is, of 
course, some long-term possibility that sanctions could be put in place on the basis of the 
standards, but that is not spelled out and would, in any event, require statutory changes. 

The second fear is that the standards are meant to create a nationwide system for aggregat- 
ing performance information on vocational education to develop, in effect, shorthand indica- 
tors of program effectiveness. This question was seldom explicitly discussed during the 
Federal-level interviews. Indeed, it came up, for the most part, during conversations with 
state and local administrators. But, by inference. Federal officials cannot seriously be con- 
sidering such a result. They demonstrate a firm grasp of the diversity of vocational educa- 
tion programs at the local level. Their reliance on periodic renewals of the National 
Assessment of Vocational Education and special studies by the NCRVE and other institu- 
tions reveals that Congress does not expect to evaluate its investment via a single national 
data system. It is hard to believe they envision a system in which information on the effec- 
tiveness of homemaking programs offered at a local high school would be combined with data 
on electronics training at the local community college to provide an aggregate picture of ef- 
fectiveness. 
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Nevertheless, despite the support for improving program performance at the local level, 
members of Congress and senior committee staff are concerned about the quality of local pro- 
grams and imcertain about whether local programs are implemented with due concern for 
the needs of local students and employers. This has become an issue based on business lead- 
ers' complaints about the quality of high*school graduates across the board and on 
Congressional staff members^ perceptions that some local managers have become bureau- 
cratic, deadened by routine and more concerned with protecting staff jobs than with meeting 
local labor market needs. These concerns are based on impressions and anecdotal evidence 
during testimony. Nobody knows exactly how widespread these problems really are, but 
Congressional supporters of vocational education have found themselves on the defensive, 
with no strong evidence to support their positions. 

The Perkins Act requirements, in this sense, were meant to reassure critics that the legisla- 
tive supporters of vocational education were serious about quality, and that lax or self-indul- 
gent program administration would not be tolerated. But members of Congress did not want 
to establish themselves as remote, ultimate judges of local processes or local outcomes. They 
wanted, instead, to create a framework of processes that would guarantee that local qxiality 
issues received due attention at the local and state levels. "Program effectiveness has to be 
evaluated at the local level," said one Federal official. "Federal officials should not try to use 
a can opener to examine local programs." Or, put another way, neither analysts nor policy- 
makers should be too quick to open the Pandora's box of accoimtability, based on a single na- 
tional data system, imless they are relatively confident they will know what to do with what- 
ever emerges. 

Is there, then, any role for nationally aggregated data on the effectiveness of vocational edu- 
cation programs? We believe there is, and the following section addresses this topic. 
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5. IMPLICATIONS FOR ACCOUNTABBLnY IN VOCATIONAL 

EDUCATION 



Members of Congress and senior federal officials cannot make decisions about local program 
quality based on standard national data. Does this mean they can have no role in overseeing 
the performance of local vocational education programs? Based on our research, the answer 
is no. Clearly, members of Congress and senior federal officials have a stake in the perfor- 
mance of vocational education and the capability to influence performance. Their influence 
lies in the leverage implicit in legislative authority. This section explores the character of 
that leverage. 

In our interviews with leaders of the Congressional staffs and federal officials responsible for 
vocational education, most were completely realistic about the relatively small size of the 
federal investment, and about the fact that federal, state, and loca' efforts are mixed differ- 
ently in almost every school district. As a senior executive branch official said, *Tou have to 
understand that federal money is a very small part of total spending on vocational education. 
All the federal government can really do with its money is to tweak ongoing state and local 
programs, to move their objectives and performance toward national objectives. Federal 
money can influence local programs to take the needs of special populations more seriously. 
And, since some people in Congress think states and localities are too lax about quality, the 
feds can establish a process that puts pressure on and might lead to some improvement." 
From interviews at the state level, this reading of Congress and senior federal officials seems 
to apply with eqxial effect to state legislators, governors, and commissioners of education. 
Although members of Congress would like to know the precise effects of their actions, gener- 
ally they do not expect to be able to micromanage vocational education or to get pure, unal- 
loyed measures of the results of federal spending. They do, however, need some basis for con- 
fidence that the programs they support are creating definite benefits. 

Congress, and often legislatures, are forums for the representation of the interests of discrete 
geographic areas. One of the m^or responsibilities of any legislator is the creation of bene- 
fits for the constituents in the district he or she represents. In assessing the benefits of a 
program or policy, therefore, legislators turn first to their own experience with the program 
at the local level, and to the opinions of key constituents, such as business and labor leaders, 
other public officials, and educators. If legislators believe a program is creating benefits for 
their constituents, they are less likely to worry about either ideological concerns or nation- 
wide or statewide evaluation effects. ^ Similarly, if they believe a program is useless to their 
districts, they are unlikely to pay much attention to it, regardless of its apparent national 
benefits, and may, in fact, oppose it if they believe an alternative program will better serve 
their own constituents. 



A highly visible demonstration of this phenomenon at work was provided in 1973 by Gerald R Ford then 
Mmonty Leader ofthe U.S. House of Representatives. Throughojt his career, Ford opposed Federal aid to the arts 
as an mduigence. He surpnsed the entire House by supporting a 1973 extension ofthe National Endowment for the 
Arts because, as he explamed, he saw how a massive outdoor mobile commissioned by the Endowment helped re- 
store business activity in his hometown, Grand Rapids. 
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Members of legislative committees assume obligations that differ from those of their col- 
leagues. Committee members, ideally, become experts on the legislation under the jurisdic- 
tion of their committee; frequently they also become advocates for the legislation, and they go 
to great effort to make certain it provides the benefits it is intended to deliver, as effectively 
as possible, to as many recipients and districts as possible. In meeting these obligations, 
committee members must be responsive to the opinions of their colleagues about the value of 
the committee's programs. If sufficient non-committee legislators believe a program is creat- 
ing important benefits for their own constituents, support for continuation and enhancement 
of the committee's legislation is that much easier to obtain. Conversely, the perception that a 
particular piece of legislation benefits only a few places makes finding a majority to support 
it that much more difficult. ^ 

Evidence about program performance, therefore, becomes an important consideration for 
committee members. In asking for such evidence, committee leaders normally want infor- 
mation that will impress their colleagues. But the researchers and analysts to whom they 
turn for help often have very different views about what evidence is germane, and at what 
level of aggregation it is most useful. Committee leaders want information that will convince 
individual legislators that a program is valuable to their districts. Poli<y analysts and data 
experts, however, often develop information that is valid only if generalized to the nation as a 
whole, based on objective measurement, and free of opinion. Many legislative leaders do not 
consider such data good substitutes for local evidence, and they are no substitute at all for lo- 
cal opinion. In short, legislators are much more interested in the aggregation of support than 
in the aggregation of data. 

In the case of the Perkins requirements on performance standards. Congress asked states to 
develop their own standards and to implement them such that local businesses, parents, and 
program operators could identify needs for improving their own programs. In many ways, 
the intent of the new 1990 performance standards restores the Federal interest in accoimt- 
ability to Senator Robert Kennedy's 1965 vision. According to interviews with Congressional 
staff, the legislative drafters obviously hoped that the data and subsequent improvement ef- 
forts would lead to greater satisfaction with vocational education. Ideally, more members of 
Congress would hear more positive things about vocational education; committee leaders, in 
consequence, would experience less difficulty garnering support in a time of both tight bud- 
gets and an apparent decline in Federal support for vocational education. Many researchers 
automatically interpreted the requirements, initially, as a demand for program operators in 
all localities to report standard information according to set formats. Those data would be 
aggregated and reduced to nationally representative statistics. 

Legislative sponsors of vocational education are not entirely indifferent to such national ag- 
gregate data. A simple, clear statement of average national effects is reassuring to leaders 
who must endure criticism, and it is useful in dealing with journalists and policy analysts 
who deal in generalizations and coefficients. But national averages are no substitute for lo- 
cal performance.^ National averages can mask critical local differences: positive average re- 



in 1981, funding for Federal job training was cut almost in half when the existing Comprehensive Employment 
and Training Act (CETA) legislation was replaced by JTPA because, in part, many legislators had become convinced 
that large amounts of CETA money were misused at the local level. Likewis**, catastrophic health coverage for the 
elderly was eliminated within six months of enactment in 1990 as a result of a nationwide barrage of complaints 
from the elderly about its costs. 
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suits can obsciire local failures that threaten future support, and negative average results 
can embarrass legislators who would, on the basis of their own constituencies' experience, be 
strong supporters. The same is true of aggregate results that show no effects because they 
average out the effects of good and bad local programs. If such results are used to justify the 
program's elimination, they can harm all the intended beneficiaries. Services can be with- 
drawn from students who are benefiting, and students who experienced poor programs do not 
get the improved services they need. 

POLITICAL ACCOUNTABILITY AND PROGRAM IMPROVEMENT 

Legislators* preoccupation with the aggregation of support is not evidence of indifference to 
outcomes. Legislative leaders know that; a program can get stable support only if it is doing 
something positive. DecHnes in support are political *^arket signals." Something needs to 
be done to change the program so that the local constituencies to which legislators respond 
(in the case of vocational education, parents, employers, students, and local government offi- 
cials) will be more satisfied with it. 

Legislative leaders typically assume that support for a program will be high if local con- 
stituencies have the evidence necessary to hold program operators accoimtable for meeting 
their needs.^ The legislator's natural response to evidence of dwindling local support is to 
intervene in the local process to ensure that the needs of key constituencies are met. In some 
areas (e.g., the rights of the handicapped or of racial minorities) legislators intervene by ex- 
posing local officials to legal process or formal enforcement action. But for constituencies 
that can fend for themselves (e.g., employers), legislators typically intervene by providing 
information that stimulates local ferment.^ 

If information to help employers and parents assess the value of the training students re- 
ceive is available at the local level, the accoimtability process is strengthened and the legisla- 
tive process is made that much easier. Local educators cannot afford to be inattentive to 
changes in job requirements if local employers and parents have realistic information on both 
job change and the relevance of local program offerings. 

For members of Congress, this reliance on local accountability processes is not naive, nor is it 
less honest or sophisticated than approaches that emphasize nationally representative statis- 
tics on standard outcome measures. In a system in which every local school system has its 
own distinctive institutional structure and its own approach to vocational education, aggre- 
gate indicators may create pressures for inappropriate regulation and standardization of 
practice. Falling national average performance can lead to efforts to design uniform national 
interventions, that is, standards, quotas, or mandates of one kind or another. These are 
seldom either sophisticated or effective. Systemwide solutions usually ignore the diversity of 
local goals, problems, and capabilities. They may or may not respond to local needs and may 
or may not affect the outcomes that most concern legislators — ^levels of local support for the 
program. In a diverse and decentralized nationwide system, legislators' reliance on local ac- 



^ThiB is the essence of the political accountability process defined by Henry Levin, op. cit. 

*See, for example, Paul T. Hill, •Evaluating Education Programs for Federal Policymakers: Lessons ftx>m the NIE 
Compensatory Education Study," in John Pincus (ed.) EducaUonal Evaluation in the Public Policy SetUnsi, (RAND 
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countability and program improvement processes not only is defensible, it may be the only 
approach that has any chance of working. 

THE PLACE FOR AGGREGATE INFORMATION 

Is there any need for broadly aggregated information? The answer is a definite yes. The 
most serious weakness of reliance on local accountability processes is that local employers, 
parents, and educators may not understand the demands that national or international 
competition soon will impose on the students in their own communities. They may noc 
demand that local programs meet high enough standards, or they may not aiderstand how 
much job skill requirements will change in the future. For these reasons, the ^ocal political 
process cannot rely entirely on local information. Th^re is a need for broader projections of 
national and regional economic trends, and for some bases on which to compare local 
graduates' skills with those of students from other areas. 

But to say that local people need national bases of comparison is not to conclude that there is 
a need for official national standards, or that such standards become the basis of nationally 
administered rewards and sanctions for local providers. To the contrary, as we have seen in 
general elementary and secondary education, centrally administered accountability processes 
can weaken the influence of local stakeholders, parents, employers, and students.^ To the 
degree that educational administrators regard themselves as members of a hierarchical bu- 
reaucratic system that will reward and punish them according to fixed rules, they will focus 
on pleasing their organizational superiors. 

The purpose of extra-local data, statewide or national comparisons, for example, must be to 
strengthen the local accountability process, not to replace it. Parents and employers need to 
be able to ask why local students do not meet national standards, or why local programs have 
not changed to accommodate changes in technology or manpower demand. Aggregate data 
should therefore be available, published in ways that local parents, employers, government 
officials, and citizens readily can use and that local educators cannot ignore. But compara- 
tive data must not be handled in ways that lead local stakeholders to believe that high offi- 
cials — legislators, governors, state or federal administrators — ^have assumed responsibility 
for program improvement. To preserve the local accountability process — to ensure that the 
people with the greatest stakes in the performance of vocational education are well in- 
formed — aggregate data and national standards must be provided as a resource for local ac- 
tors, not as an alternative to local action. 

FEDERAL AND STATE ASSISTANCE 

The foregoing does not mean to argue that the existing local accoxmtability processes for vo- 
cational education are fully adequate. They need to be strengthened in the ways that the 
Perkins requirements prescribe. Though it is clear that many local vocational education in- 
stitutions have close working relationships vnth parents, employers, and related higher-level 
institutions, many apparently do not. As some state and local administrators have told us, it 
still is possible for high schools to continue programs that have low enrollments and no con- 
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nections to jobs, or for programs to prepare students for low-level jobs when better jobs are 
available. As some postsecondary education administrators have admitted, vocational edu- 
cation professors are as likely as other tenured faculty to teach from yellowed notes and to 
ignore important economic and technical developments. There is a need for stronger local ac- 
countability processes, with the characteristics described above in the section on performance 
standards and accountability. 

The Federal and state governments can do a great deal to enable local educators, employers, 
and parents to create such processes. Higher-level governments can collect data for local ju- 
risdictions and publish them at the local level for use by students, parents, and employers. 
They can sponsor development of technical standards for traixiing in rapidly changing techni- 
cal subjects, or in skills for which the labor market is national rather than local. National 
studies such as the National Assessment of Vocational Education and the NCRVE research 
program can help to clarify employers' expectations for workers in entry-level jobs and can 
help distinguish among the academic, technical, and attitudinal dimensions of work pre- 
paredness. Federal research institutions can provide examples of particularly effective 
methods for teaching especially difficult skills, or for teaching disadvantaged students. All 
those kinds of information help local educators, employers, students, and parents construct 
reasonable expectations for local program performance and increase performance pressure on 
local vocational education providers. 

More general performance standards, such as student placement rates or average post-train- 
ing wages, are more problematic. Taken out of the context of local economic conditions, such 
national standards and averages may be inappropriately rigorous or lax. A 70 percent 
placement rate for a vocational education program in a tight labor market may reflect very 
poor performance, while the same rate may be sensational in a depressed area. Even when 
such standards are adjusted for local conditions, they still concentrate local program opera- 
tor's attention ol satisfying external, not local constituencies. 

IMPLICATIONS 

This reasoning implies that: (1) data aggregated at the state and national levels on the 
performance of students in vocational education are inherently untrustworthy as guides to 
national action; (2) local program performance cannot be improved by such data; and (3) re- 
liance on aggregate data as the sole accoimtability mechanism may dilute — even distort — 
promising local accountability procedures groimded in satisfying the immediate clients of 
vocational education, namely parents, students, and employers. Judiciously developed 
performance information on carefully selected samples and subsets of vocational education 
programs can serve as an indicator of the health of the national enterprise, but its 
implications for federal action are not direct or obvious. What Congress really needs to know 
is whether state and local providers have the resources and incentives necessary to solve 
local problems. 

This argument does not pretend that local accountability processes in every locality are ade- 
quate, or that Federal and state supervision of program quality are imnecessary. On the 
contraiy, in many local school districts, accountability procedures are fragmented, diffuse, 
and difficult to define; in every locality, oversight by Federal and state funding agencies is 
essential. But from on high, neither the states nor the Federal government can guarantee 
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program accountability to local needs. By definition, local accountability has to be put in 
place at the local level. The Federal role becomes one of encouraging procedures to take local 
needs into account; the state role is one of monitoring local procedures and providing advice 
and technical assistance as required. 

In line with this general argument, we believe two additional caveats are in order: 

* Local performance outcome data should supplement, not supplant, processes of consulta- 
tion and the exercise of administrative judgment. 

• Vocational administrators should not abandon traditional monitoring in favor of "automat- 
ed** accountability processes. 

Data Should Supplement Local Processes 

All educational outcomes are products of multiple causation. Test scores, graduation rates, 
dropout rates, and student placement results obviously reflect on the quality of educational 
services. But they also can be strongly affected by other factors independent of the quality of 
training and instruction students have received. 

The outcomes of greatest interest in vocational education are affected by local labor market 
conditions, as well. Vocational education students' job placement rates, hours of work, in- 
come, and rates of career progression can be affected profoundly by the demand for particular 
kinds of labor, which is determined in turn by the current level and composition of local eco- 
nomic activity. 

These complications do not rule out the use of hard outcomes data. But they do illustrate the 
many ways in which such data can foster misiinderstanding of program performance. 
Outcomes? data are indispensable, but only when they can be interpreted in light of supple- 
mentary information about student characteristics and prior educational preparation, the 
quality of educational programs that complement vocational education, and local labor mar- 
ket conditions. In light of such contextual information, programs can be assessed fairly and 
performance deficiencies can be identified readily. 

Without contextual information, vocational educators are forced to do everything possible to 
raise their outcome scores. This promotes "gaming" to increase scores (e.g., by creaming the 
student population or finding ways to discount the outcomes of less able students). It also 
misleads policymakers, frequently causing them to castigate local providers whose programs 
are, in fact, performing reasonably well in light of local conditions. 

Monitoring VerSkis "Automated** Accountability 

Testing and formal outcomes measurement are not cheap, but they are far less labor-inten- 
sive than direct program monitoring and careful accreditation of institutions. During a pe- 
riod of fiscal stringency, many states are tempted to cut their staffs and to abandon direct 
oversight in favor of automated systems of measurement. 

These systems can be m^or assets, both for state and national officials responsible for voca- 
tional education, and for actors at the local level. Only such ''hard'* measures can provide 
clear comparative information about the priorities and average outcomes attained by differ- 
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ent localities and programs. But, as suggested above, hard outcomes data alone can lead to 
false attributions of causality and can mislead policymakers. State and federal officials may 
overlook crucial local conditions and reward or punish programs whose outcomes are largely 
determined, for good or ill, by imcontrollable local circumstances. 

Assessment and reward systems that are insensitive to local conditions can distort local pro- 
gra mmin g and practice. The long-run consequences can be extremely negative: Local 
providers of vocational education can come to de-emphasize the processes whereby they tailor 
vocational education to the needs of local students and employers. As we have seen in other 
msyor federal education programs, such as ESEA Chapter I, this leads providers to separate 
and distinguish the services they provide from services funded by other sources. 

As state-funded or federally funded services become more distinctive and auditable, they also 
do less to complement and strengthen the basic educational programs provided at the local 
level. This process ultimately reduces the productivity of the educational system as a whole, 
because students' performance in any one area is affected by the quality of their preparation 
in all areas. 

Becatise job competencies, placement rates, and other *liard" outcomes of vocational educa- 
tion are joint products of vocational and general education, vocational education cannot af- 
ford to employ an accountability system that ultimately weakens overall local educational 
performance. Vocational education cannot produce positive outcomes if students' general ed- 
ucation preparation is abysmal. In education, a falling tide lowers all boats: local citi2ens, 
employers, and civic leaders faced vrith a poorly performing education system seldom make 
fine distinctions among programs. Similarly, national business leaders and others who decry 
the poor level of student preparation are not able to disentangle vocational from other forms 
of education: They know only that the system is failing them. 

When faced with complaints about system failure, top-level policymakers (Congress, state 
legislatures, presidents, governors) demand accountability. But they mean systems account- 
ability, not separate auditing of the effects of different programs. If the response to the call 
for accountability is to further separate educational programs from one another, system per- 
formance will continue to decline, with increasingly negative consequences for vocational ed- 
ucation's reputation and public support. 

The bottom line is that vocational education must become an integral part of the local educa- 
tional system, and its accountability system must emphasize meeting local needs. Standard- 
ized outcome measures can strengthen the local accountability process if they give local 
parents, employers, and commimity leaders objective information about local program 
outcomes. But if they are used in ways that weaken local accountability, national data bases 
on local program outcome ultimately can impair local program performance and further 
damage the local and national reputation of vocational education. 

To conclude, a national data base does not solve the problem of accountability in vocational 
education, and excessive reliance on it can threaten the performance of local programs. It is 
an example of a tool that determines its own use. In order to avoid the consequences of a na- 
tional outcomes data base, false advertising about its value must be avoided. A national data 
base that is expected to be too much will be a Pandora's box. If built, someone surely will 
open it, and once opened, like every other Pandora's box, it will turn out to be not a gift, but a 
snare. 



6. CONCLUSIONS 



Some things can be put back into Pandora's box. Congress' need for assurance about the 
proper use of federal funds does not require a national data system based on uniform report- 
ing of student outcomes. National concerns about program accountability need not force vo- 
cational educators to measure their results via a small standard set of student outcomes, or 
to make program initiation and termination decisions according to a limited set of uniform 
national criteria. The perverse effects suffered by other federal education programs — stan- 
dardizing of services across localities with widely varying needs and encouraging localities to 
provide only those services and to serve only those students that make them look good on na- 
tional evaluations— can be avoided. This message applies equally to Section I and to any 
other federal program that intends to assist an activity for which states and localities take 
primary responsibility. 

But that does not mean that Congress or the national government should abstain from any 
effort to learn how federal funds are spent or whether states and localities are making con- 
scientious efforts to use them well. Periodic national evaluations and special-puipose studies 
can both report on the breadth and diversity of vocational education services and call atten- 
tion to areas in which programs are failing and new national priorities or incentives are 
needed. States and localities also have a part in reassuring Congress that it is investing in a 
serious and useful effort; they can do so by making explicit reports on their strategies for 
identifying troubled local programs and providing real assistance to them. 

In any domestic program area, Congress opens the Pandora's box of national program evalu- 
ation because it has reason to doubt that recipients are taking serious account of the pur- 
poses for which federal funds were appropriated. In the nearly 30 years since enactment of 
the first major program of federal aid to education. Congress has become skeptical of pro- 
grams that are routinized, comfortable, and bureaucratic. Its demands for accountability are 
efforts to ensure that federal funds are helping solve real problems. The challenge posed by 
congressional actions such as the Perkins Act performance standards and measures require- 
ments is how Congress can get the assurance it seeks without distorting program activities 
and priorities. 

Yet, the challenge posed by Congress is not well met by systems or regulation and measure- 
ment that impose new constraints on local problem solving. As the Department of Education 
has held since Perkins was enacted, the challenge can be met only by serious efforts to im- 
prove programs at the local level. Many states have maintained this capacity, and some are 
treating the Perkins Act requirements as an occasion to strengthen and renew their over> 
sight and assistance relationships with local vocational education providers. Some, unfortu- 
nately, are going the other way, cutting their own capacities to visit local programs and pro- 
vide advice and assistance when needed, and substituting cheaper testing programs. Such 
programs may be constructive parts of a more complete assessment and assistance strategy, 
but taken alone they can do little to improve programs and much to reduce local flexibility 
and problem-solving capability. Inevitably, they call attention to what is already well 
known — that vocational education (and, for that matter, education, in general) is not meeting 
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the needs of students and employers in most msyor metropolitan areas. This fact requires a 
serious improvement effort, not documentation. 

Congress has a part to play in helping put the destructive aspects of accountability back into 
Pandora's box. It should match its reasonable position about measurement with a more real- 
istic approach: support of state oversight to identify localities that are failing their students 
and investment to help (or shame) those localities toward more effective services. It should 
limit r^ulations that focus state education agency efforts on compliance and adopt policies 
that direct states toward helping schools and districts that lack the capacity or the will to 
serve students well. But the ultimate burden is on the states, which must decide how local 
vocational education programs can be made more responsive to the needs of students and the 
local and state economy. If states show that they are taking this challenge seriously, they 
will surely satisfy the need for reassurance that led Congress to enact the Perkins require- 
ments. 
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